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I.  INTRODUCTION 


The  Army  Science  Board  (ASB)  Ad  Hoc  Subgroup  on  Testing  of  Electronic 
Systems  was  established  at  the  request  of  the  Assistant  Secretary  of  The 
Army  (Research,  Development  and  Acquisition),  whose  letter  to  the  ASB 
Chairman  included  these  statements  (cf .  Appendix  A)  : 

"As  increasingly  sophisticated  Army  systems  are  developed,  the 
testing  and  evaluation  of  these  systems  during  the  acquisition  cycle 
become  more  challenging.  Particularly  difficult  is  the  testing  of 
the  C^I  and  computer-based  portions  of  these  systems  in  realistic 
battlefield  environments  that  include  anticipated  levels  of  input- 
output,  system  software  loading,  electronic  threats,  and  maintenance. 
Present  approaches  include  simulation  and  field  exercise.  The  expense 
of  elaborate  testing  must  be  weighed  against  the  risk  of  detecting 
potential  system  failure  mechanisms  and  operational  difficulties  only 
after  development  and  fielding. 

The  ASB  Panel  should  examine  the  overall  facets  of  this  subject, 
specifically  addressing  the  following: 

1.  Are  Army  concepts,  plans  and  equipments  adequate  for  the 
testing  of  modem  C^I  and  computer  based  systems? 

2.  What  changes  should  be  made,  if  any? 

This  investigation  should  include  an  assessment  of  relevant  testing 
facilities,  including  TRI-TAC's  Joint  Test  Facility  at  Fort  Huachuca, 
the  Automated  Systems  Test  Bed  at  Fort  Hood,  and  plans  for  the  Modular 
Automated  Integrated  Systems  Interoperability  Test  and  Evaluation 
(MAINSITE)  System.  The  adequacy  of  facilities,  test  equipment,  pro¬ 
cedures  and  plans  to  support  testing  of  ASAS,  PLRS,  and  TACFIRE 
should  be  addressed.  In  addition,  suggestions  for  satisfactorily 
operationally  testing  future  software  systems  would  be  appreciated." 

The  initial  meetings  of  the  Subgroup,  devoted  primarily  to  overviews  of 
test  organizations,  facilities  and  approaches,  were  held  in  the  Washington 
area.  Subsequently,  to  engage  in  discussions  with  personnel  directly  in¬ 
volved  in  Army  testing,  visits  were  made  to  the  Combined  Arms  Test  Facility, 
Fort  Hood;  The  Missile  Command,  Redstone  Arsenal;  the  White  Sands  Missile 
Range;  and  the  Electronic  Proving  Ground,  Fort  Huachuca. 

Appendix  B  contains  brief  summaries  of  the  severai^lubgroup  meetings, 
along  with  a  listing  of  all  presentations  given  at  the  meetings.  As  noted, 
in  addition  to  presentations  by  various  Army  testing  agencies /activities, 
OSD/OUSDR&E  perspective  relative  to  testing  was  furnished  by  the  Deputy 
Director  for  Tactical  Air  and  Land  Warfare  Systems;  and  a  series  of 


presentations  was  requested  from  organizations  involved  in  U.S.  Navy  testing 
to  provide  information  that  could  be  used  to  make  a  limited  comparison  of 
the  testing  approaches  of  the  two  Services. 

For  reference  in  this  report,  Appendix  C  contains  viewgraph  prints 
from  a  presentation  made  by  Mr.  J.  P.  Tyler  (DAMA  —  Policy,  Plans,  Manage¬ 
ment  Division),  entitled  "An  Overview  of  Army  Materiel  Testing";  it  provides 
definitions  of  types  of  tests  and  related  documentation,  along  with  outlines 
of  organizational  relationships.  Appendix  D,  prepared  by  LTC  Dennis  O’Connor, 
includes  more  detailed  information  relative  to  the  missions  of  all  major 
Army  test  facilities;  a  glossary  of  testing  terms;  and  a  series  of  outlines 
showing  the  progression  of  types  of  tests  through  the  various  facilities. 

Also  for  reference  in  this  report,  Appendix  E  contains  viewgraph  prints 
from  a  presentation  by  BG  Jerry  Max  Bunyard,  PATRIOT  Project  Manager, 
entitled  "PATRIOT  Project  —  Lessons  Learned".  Appendix  F  includes  memoranda 
prepared  by  members  of  a  PATRIOT  Program  Review  Panel  established  by  the 
Assistant  Secretary  of  the  Army  (RD&A) ,  incorporating  suggestions  relating 
to  future  Army  development  and  testing. 

Appendix  G  includes  the  Interim  Report  of  the  Subgroup,  as  presented 
to  the  Army  Science  Board  meeting  on  March  16. 

The  members  of  the  Subgroup  would  like  to  express  their  thanks  to  the 
Commanding  Generals  of  the  various  installations  visited,  and  to  the  presenters 
identified  in  Appendix  B.  The  cooperation  and  interest  of  all  participants 
made  the  investigation  a  rewarding  experience  for  the  Subgroup. 


II.  SUMMARY  OF  FINDINCS  AND  RECOMMENDATIONS 


The  primary  questions  for  consideration  by  the  Ad  Hoc  Subgroup  on 
Testing  of  Electronic  Systems  were  stated  as  follows: 

1.  Are  Army  concepts,  plans  and  equipment  adequate  for  the 
testing  of  modern  C^I  and  computer-based  systems? 

2.  What  changes  should  be  made,  if  any? 

To  the  extent  of  the  findings  of  this  report,  it  is  the  opinion  of  the 
Subgroup  that  the  response  to  the  first  question  must  be  in  the  negative. 
Additionally  it  is  felt  that  the  shortcomings  currently  associated  with 
"testing"  cannot  be  considered  in  isolation  from  more  general  problems  of 
the  system  acquisition  process.  As  a  consequence,  many  of  the  recommenda¬ 
tions  developed  in  response  to  the  second  question  are  far-reaching  and 
will  be  difficult  to  implement.  The  problems  outlined  in  the  findings  are 
fundamental,  however,  and  will  not  be  solved  without  substantive  action. 

The  primary  findings  and  recommendations  focus  on  the  following: 

1.  The  need  for  much  stronger  concept  definition  and  more  orderly 
design/testing  in  the  early  developmental  phases  of  Army  system  acquisition; 

2.  The  need  to  include  software  testing  as  an  integral  part  of  total 
system  test  plans  using  state-of-the-art  software  verification  and  valida¬ 
tion  tools; 

3.  The  need  to  introduce  parallelism  in  the  Army's  currently  serial 
development/testing  process,  with  special  reference  to  parallel  development 
of  the  computer-based  test  tools  required  for  evaluation  of  software¬ 
intensive  systems; 

4.  The  need  to  strengthen  the  post-DSARC  III  testing  and  follow-on 
evaluation  (FOE)  of  systems  as  they  move  from  DT-II/OT-II  to  full 
production  in  order  to  combat  the  effects  of  employing  prototype  hardware 
and  immature  software  for  DT-II/OT-II  tests; 

5.  The  need  for  much  more  coordination  of  planning  for  electronic- 
system  test  facilities  within  the  Army  with  regard  to  both  development 
and  usage; 

6.  The  need  to  strengthen  the  extent  and  fidelity  of  interoperability 
testing. 
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7.  The  need  to  improve  the  Army's  ability  to  provide  technical  conti¬ 
nuity  and  corporate  memory  within  programs  and  from  program-to-program  to 
combat  the  effects  of  long  program  lifetimes  and  organizational  boundaries. 

The  findings  and  recommendations  of  these  seven  areas  are  summarized 
on  pp.  5-11  and  discussed  in  more  detail  in  Sections  III-VII. 

It  is  recognized  that  considerations  of  over-all  Army  organization 
for  testing  are  beyond  the  scope  of  this  Subgroup;  however,  a  few  comments 
would  appear  to  be  in  order.  The  relevant  aspects  of  Army  organization/ 
facilities  are  outlined  in  Appendices  C  and  D;  as  noted  in  Section  VII  of 
this  report,  the  indicated  organizational  structure  seems  complicated  and 
cumbersome.  The  assignments  of  responsibility  for  the  various  facets  of 
test  policy,  test  management,  test  implementation,  and  test  evaluation 
appear  in  some  cases  to  be  fragmented  and  inefficient,  and  tend  to  amplify 
technical-continuity  difficulties  outlined  in  the  findings  and  recommenda¬ 
tions  and  discussed  in  Sections  III,  VI  and  VII.  It  is  suggested  that  a 
reassessment  of  the  Army's  organization  for  testing  may  be  in  order,  with 
a  view  toward  increasing  efficiency  through  centralization  of  responsibility. 


FINDINGS 


With  the  present  approach  to  development,  some  systems  have  entered 
advanced  phases  of  operational  testing  prior  to  the  identification  of 
major  design  faults;  in  some  cases,  problems  that  have  occurred  in  opera¬ 
tional  tests  are  directly  traceable  to  shortcomings  in  basic  system  concepts. 
Primary  emphasis  has  been  given  to  the  meeting  of  established  operational 
test  (and  ASARC/DSARC)  schedules,  with  inadequate  attention  to  actur1. 
design  status  and  readiness  for  testing;  in  point  of  fact,  adequate 
"visibility"  relative  to  design  status  has  in  many  instances  been  unavail¬ 
able  prior  to  the  initiation  of  operational  testing.  The  inflexibility 
of  operational  test  schedules  has  been  counterproductive,  leading  to 
subsequent  prolonged  program  delays  and  increased  program  costs. 


RECOMMENDATIONS 

1.  Additional  effort  should  be  devoted  to  the  concept  definition/ 
concept  evaluation/advanced  development  phases  of  system  development; 
additional  consideration  should  be  given  to  early  system  simulation  and  to 
tradeoffs  among  performance  and  reliability/availability /maintainability; 
in  this  connection,  Army  in-house  capability  as  "wise  buyers"  should  be 
improved . 


2.  During  engineering  development,  a  philosophy  of  incremental  step- 
by-step  design/testing  should  be  employed;  additional  emphasis  should  be 
placed  on  hardware  and  software  subsystem  testing  and  on  hardware/software 
integration. 

3.  Additional  attention  should  be  given  to  the  explicit  understanding 
of  design  status  at  all  times,  with  formal  reviews  (for  both  hardware  and 
software)  throughout  engineering  development;  although  planning /requirements 
for  operational  tests  (OT)  should  be  established  early  in  the  development 
process,  development  tests  (DT)  should  in  all  cases  be  completed  and 
evaluated  prior  to  the  related  phase  of  OT;  discovery  of  major  design 
faults  during  DT  should  result  in  redesign/retest  prior  to  OT. 

4.  It  should  be  recognized  that  additional  (higher-than-normal) 
funding  in  early  program  stages  —  with  effective  program  management  — 
can  be  expected  to  lead  to  reduced  life-cycle  costs  and  shortened  time 
scales. 


FINDINGS 


Relative  to  software  design  and  testing,  it  has  not  been  understood 
that  effective  software  design  —  to  an  even  greater  extent  than  effective 
hardware  design  —  is  dependent  upon  the  existence  of  agreed,  specific, 
properly-documented  system  requirements.  It  has  not  been  generally  recog¬ 
nized  that  techniques  for  reliable  and  comprehensive  software  testing  are 
entirely  different  from  comparable  hardware  testing  techniques.  Further¬ 
more,  advantage  has  not  been  taken  of  the  fact  that  early  software  testing 
allows  correction  of  design  flaws  at  much  less  expense  than  correction 
later,  and  often  permits  a  technically  superior  solution  involving 
architectural  and/or  hardware  changes  which  may  become  impractical  later 
in  the  development  cycle. 

RECOMMENDATIONS 

1.  For  all  programs,  additional  emphasis  should  be  given  to  early 
establishment  and  documentation  of  quantitative,  "testable"  system  require¬ 
ments,  including  environmental  and  operational  factors;  requirements/criteria 
"audit  trails"  should  be  provided  throughout  the  testing  process.1 

2.  Software  designs  should  be  required  to  be  testable  at  module  and 
subsystem  levels  (as  well  as  on  an  over-all  system  basis);  software  designs 
should  be  directly  relatable  to  system  requirements.  Program  plans  should 
include  module,  subsystem  and  system- level  software  tests  in  all  phases  of 
system  design  (with  adequate  funding  provided) ;  software  testing  should  be 
a  recognized,  required  aspect  of  formal  development  (DT)  and  operational 
testing  (OT) . 

3.  Based  on  the  system  specifications,  and  with  flexibility  in  agreed 
areas,  automated,  computer-based  test  tools  should  be  developed  to  drive 
(via  simulation  and  stimulation)  the  engineering  and  initial  production 
models  of  software- intensive  systems;  only  in  this  way  can  operational 
environments  be  suitably  represented  in  a  reproducible  fashion. 

4.  Facilities  such  as  MAINSITE2,  to  be  an  effective  DT  asset,  should 

be  designed  and  equipped  for  the  special  requirements  of  software  testing,  as 
well  as  for  hardware  testing;  lessons  learned  by  other  Army  testing  agencies, 
and  other  Services,  should  be  studied  to  assist  in  determining  MAINSITE  test¬ 
ing  requirements . 

5.  To  facilitate  cost  effective  software  testing  with  results  that  can  be 
uniformly  interpreted  and  "graded",  a  common  library  of  software  verification 
and  validation  tools  should  be  developed  and  used  on  an  Army-wide  basis;  the 
Army  should  recognize  an  opportunity  to  provide  (DoD)  leadership  in  this  regard. 


1.  Also  discussed  in  report  of  the  1980  Summer  Study  on  Statistical  Techniques 
in  Testing,  7-11  July  1980,  pp.  6-7;  and  in  report  of  the  Panel  on  Design 
of  Army  Tests,  1  May  1981,  pp.  2-3. 

Modular  Automated  Integrated  Systems  Interoperability  Test  and  Evaluation 
Systems  located  at  the  Electronic  Proving  Ground,  Fort  Huachuca. 


2. 


FINDINGS 


The  extensive  time  required  to  develop  and  deploy  systems  is  in 
part  the  result  of  the  Army's  serial  development/testing  process. 
Representatives  of  the  users  participate  in  the  initial  definition  of 
system  requirements,  and  are  responsible  for  conducting  operational  tests; 
during  system  development,  they  are  involved  primarily  as  spectators.  As 
a  result  —  and  especially  in  view  of  turnover  in  personnel  —  there  are 
discontinuities/uncertainties  in  performance  and  testing  requirements, 
especially  in  respect  to  test  environments.  Furthermore,  for  electronics/ 
software-intensive  systems,  the  development  of  requisite  computer-based 
test  tools  is  a  difficult,  long-term  task. 


RECOMMENDATIONS 

1.  It  is  suggested  that  consideration  be  given  to  a  radical  change 
in  the  development/testing  process,  in  recognition  of  the  special 
characteristics  of  software-intensive  systems;  that  the  computer-based 
test  tools  required  to  represent  the  test  (tactical)  environment  be  pro¬ 
vided  by  a  contractor  other  than  the  system  development  contractor,  in 
parallel  with  system  development.  In  this  approach,  the  testing/user 
activities  should  participate  in  the  test  contractor  design  reviews  — 
and  should  be  required  to  quantify  and  document  test  requirements. 

2.  The  indicated  test  drivers  (environment  simulators)  should  be 
developed  for  particular  programs;  however,  they  can  be  appropriately 
integrated  into  the  plans  for  testing  at  various  facilities. 

3.  The  development  of  the  test  drivers  should  be  in  accordance 
with  the  disciplines  previously  outlined  for  software  development  and 
testing  (cf.  p.  6). 


FINDINGS 


For  electronics/software-intensive  systems,  major  difficulties  in 
the  system  acquisition/testing  process  have  occurred  because  the  testing 
associated  with  initial  production  decisions  (DT-II/OT-II)  is  not  in 
general  conducted  on  true  production  prototypes;  key  system  elements  are 
typically  manufactured  under  "laboratory”  conditions  at  that  stage  of 
development;  the  software  is  usually  incomplete  and  immature.  Thus  DT-II/ 
OT-II  data  are  often  unrepresentative  of  production  designs.  At  present, 
there  tends  to  be  inadequate  recognition  of  the  foregoing  points;  as  a 
result,  there  may  be  inadequate  planning  for  design/ testing  follow-up 
during  the  period  between  the  initial  productioii  decision  (DSARC  III) 
and  the  start  of  production.  Furthermore,  follow-on  evaluations  (FOE) 
on  production  hardware  appear  to  be  scheduled  "as  needed"  —  and  may 
therefore  be  underfunded  and  limited  in  scope. 


RECOMMENDATIONS 

1.  For  all  electronics/software-intensive  systems,  additional  efforts 
should  be  devoted  to  detailed  establishment  of  relationships  between  the 
hardware/software  employed  for  DT-II/OT-II  and  the  ultimate  production 
designs . 


2.  After  OT-II,  "visibility"  relative  to  hardware/software  status 
should  be  regarded  as  critically  important;  program  check-points  and  phased 
demonstrations  should  be  scheduled  for  both  hardware  and  software  improve¬ 
ments;  the  need  for  continuation  of  hardware/software  integration  tests 
should  be  recognized. 

3.  Follow-on  evaluations  on  production  hardware  should  be  planned 
as  a  requirement  (not  on  an  "as  needed"  basis)  to  assure  adequate  funding 
and  provision  of  test  items;  for  FOE,  there  should  be  the  same  detailed 
attention  to  planning/data  collection/data  interpretation  as  that  requisite 
for  effective  DT-II/OT-II  testing;  reliability-availability-maintainability 
(RAM)  maturation  programs  should  be  regarded  as  essential. 

A.  It  should  be  recognized  that  software  designs  (which  control  the 
operational  performance  of  systems)  will  be  evolutionary;  that  hardware/ 
software  integration  testing  will  be  necessary  during  the  production  phase; 
and  that  continuing  visibility  and  adherence  to  design  disciplines  will  be 
essential. 


FINDINGS 


Relative  to  plans/facilities  for  testing,  the  Subgroup  was  favorably 
impressed  with  the  general  excellence  of  facilities,  and  with  the  compe¬ 
tence  and  dedication  of  the  technical  staffs  involved.  It  is  evident, 
however,  that  there  is  inadequate  communication  among  testing  agencies, 
and  insufficient  consideration  of  the  time  and  cost  savings  —  and  the 
improvements  in  the  understanding  of  test  results  —  that  could  be 
generated  by  better  coordination  of  testing,  and  by  development  of  comple¬ 
mentary  facilities.  The  lack  of  coordinated  planning  presumably  results 
in  part  from  the  Army’s  fragmented  organizational  alignments  for  testing; 
however,  within  the  current  organizational  structure,  improved  coordination 
of  facilities  planning  should  be  possible. 


RECOMMENDATIONS 

1.  As  one  example,  it  is  suggested  that  the  development  of  comple¬ 
mentary  plans  be  required  for  MA1NSITE  (a  C3l  DT  test  system  at  Fort 
Huachuca)  and  ATSTB  (a  C^I  OT  test  system  at  Fort  Hood)  —  although  one 
facility  is  controlled  by  TECOM,  and  the  other  by  FORSCOM/TRADOC.  The 
indicated  test  systems  require  high  levels/rates  of  expenditure,  for  tests 
of  the  same  C^I  systems.  Although  both  appear  to  be  justified,  it  would 
seem  that  substantial  advantages  could  be  gained  through  coordinated 
planning  and  interactive  employment  of  testing  resources. 

2.  Again  referring  to  planning  for  MAINSITE  and  ATSTB  as  an  example, 
it  is  suggested  that  other  software-oriented  agencies /organizations  become 
involved  in  a  coordinated  planning  process  to  assure  that  the  unique  require 
ments  of  software  testing  are  met  —  although  organizational  boundaries  must 
be  crossed. 

3.  As  another  example,  it  is  suggested  that  additional  coordination 
would  be  desirable  between  the  Electronic  Proving  Ground  and  the  White  Sands 
Missile  Range  relative  to  the  design/employment  of  ECM  test  systems. 

4.  More  generally,  it  is  felt  that  additional  coordination  of  detailed 
facility/test  system  planning  would  permit  substantial  cost/time  savings, 
and  that  immediate  steps  should  be  taken  to  outline  an  over-all  approach 

in  this  regard. 


FINDINGS 


In  various  presentations  to  the  Subgroup  (and  in  related  discussions) 
reference  was  made  to  interoperability  testing;  there  was  little  evidence, 
however,  of  coherent,  coordinated  over-all  planning.  Since  interoperability 
will  be  of  critical  importance  in  a  tactical  environment,  and  interference 
among  electronics-intensive  systems  may  be  a  major  factor,  the  apparent 
lack  of  specific  test  planning  for  interoperability  appears  to  be  a  major 
inadequacy.  In  this  connection  it  should  be  noted  that  interoperability 
involves  not  only  Army  systems,  but  also  Air  Force  and  other  NATO  equip¬ 
ments;  furthermore,  interoperability  tests  must  consider  interactions 
with  "friendly"  as  well  as  enemy  countermeasures. 


RECOMMENDATIONS 

It  is  possible  that  the  Subgroup  did  not  become  aware  of  the  extent 
of  planning  for  interoperability  testing;  and  it  is  recognized  that  not 
all  aspects  (not  even  all  significant  aspects)  of  interoperability  can  in 
fact  be  tested.  It  would  seem,  however,  chat  more  extensive  over-all 
planning  should  be  carried  out;  and  that  considered  decisions  should  be 
made  relative  to  the  omission  of  testing  for  reasons  of  complexity  or  cost. 
In  this  regard,  it  is  especially  important  that  analyses  and  simulations 
be  conducted  to  guide  decisions,  with  recognition  of  the  fact  that  appro¬ 
priate  complementary  employment  of  systems  can  greatly  enhance  over-all 
Army  combat  effectiveness. 
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FINDINGS 


As  a  system  evolves  from  concept  (TRADOC) ,  through  demonstration 
(contractor,  development  laboratory)  and  development  (contractor,  Pro¬ 
ject  Manager),  to  fielding  (readiness  side  of  commodity  command), 
technical  continuity  (on  the  part  of  the  government)  tends  to  exist  only 
in  an  archival  sense.  In  rare  instances,  not  due  to  systematic  process, 
individuals  may  shift  jobs  to  follow  a  system  through  this  cycle,  but 
inadequate  records  (rationale  for  past  choices,  data)  and  insufficient 
personal  recollections  tend  to  dominate  this  problem.  For  high  technology, 
complex  systems,  this  problem  is  exacerbated  by  longer  acquisition  cycles. 

Transfer  of  organizational  responsibility  at  system  milestones 
contributes  to  the  indicated  lack  of  technical  continuity.  Even  within 
the  time  span  a  system  remains  under  the  responsibility  of  one  office 
(e.g.,  PM),  however,  rotation  of  technical  personnel,  emphasis  on  sched¬ 
ules  and  costs,  and  tendencies  toward  insufficient  documentation  lead  to 
erosion  of  technical  knowledge. 

Furthermore  —  and  this  point  is  of  great  consequence  —  knowledge 
gained  in  any  given  program  is  infrequently  transmitted  effectively  to 
other  programs;  programs  tend  to  be  isolated,  with  limited  communication 
across  boundaries. 


RECOMMENDATIONS 

As  previously  noted  (cf.  p.  5)  improvement  of  Army  in-house  capabil¬ 
ities  as  "wise  buyers"  is  necessary;  in  addition,  better-coordinated  use 
of  in-house  capabilities  (by  continuity  of  assignments  and  coordination/ 
cooperation  across  organizational  boundaries)  is  essential  and  should  be 
pursued  on  a  high-priority  basis.  As  a  further  action,  it  may  be  appro¬ 
priate  to  consider  an  approach  employed  by  other  Services:  to  augment 
in-house  capabilities  by  establishing  a  continuing,  stable  relationship 
with  a  non-profit  organization  (e.g.,  a  Federal  Contract  Research  Center 
or  a  hardware/software-oriented  university  laboratory) .  Advantages  in 
terms  of  "corporate  memory"  and  "transmittal  of  culture"  from  one  program 
to  another,  and  from  current  programs  to  future  programs,  could  be  highly 
significant . 


III.  MANAGEMENT  OF  TESTING;  RELATED  DEVELOPMENT  ISSUES 


In  examining  the  question  of  testing  of  complex  electronic  equip¬ 
ment,  several  related  issues  need  to  be  addressed.  Most  importantly, 
it  is  evident  that  the  primary  concern  should  not  be  testing  alone, 
but  rather  the  general  subject  of  Army  actions  necessary  to  improve 
the  reliability,  maintainability,  performance,  and  cost  effectiveness 
of  its  equipment.  Improved  testing  is  only  a  small  and  often  misunder¬ 
stood  part  of  the  answer.  For  example,  no  amount  of  testing  can  correct 
an  improperly  specified  system,  a  poor  design,  a  software  compatibility 
problem,  an  ECM  problem,  or  a  poor  maintenance  concept. 

One  objective  of  operational  testing  is  to  uncover  problems  that 
were  missed  during  the  design  phase;  however,  these  should  be  the 
exceptional  cases  rather  than  the  rule.  Too  frequently,  test-and-fix 
is  used  as  a  crutch  for  a  poorly  considered  design.  This  is  time 
consuming,  always  costly,  and  sometimes  the  implementation  proves  to 
be  impractical.  Much  greater  emphasis  needs  to  be  placed  on  the  early 
design  phases  of  programs . 

Basic  to  any  procurement  is  a  realistic  system  specification  (which 
recognizes  technological  bounds)  establishing  the  equipment  performance 
requirements,  and  the  operating  environment.  The  environment  should 
include  not  only  the  usual  physical  aspects,  such  as  temperature,  humid¬ 
ity,  vibration,  etc.,  but  also  the  ECM  threat,  reliability  requirements, 
compatibility  and  interoperability  (both  hardware  and  software)  with 
other  friendly  equipment,  the  skill  level  of  the  operation  and  mainte¬ 
nance  crews,  logistic  support  levels,  etc.  The  Army’s  understanding  of 
these  requirements  should  be  reflected  in  specifically  tailored  system 
specifications.  This  is  a  fundamental  starting  point  for  any  procurement, 
and  its  importance  cannot  be  overemphasized.  Untailored  system  specifi¬ 
cations,  based  on  boilerplate  military  standards,  more  often  than  not 
result  in  equipment  which  does  not  meet  the  needs  of  the  Army.  Without 
a  tailored  specification,  critical  performance  deficiencies  will, 
despite  a  rigorous  test  program,  remain  largely  undetected  until  the 
equipment  enters  the  field.  At  that  point  in  time,  the  cost  to  fix 
and  retrofit  often  becomes  prohibitive. 

The  source  selection  process  is  another  area  that  needs  Army  top 
management  attention.  The  services  have  long  given  lip  service  to 
reliability,  availability,  maintainability  (RAM)  and  life  cycle  costs, 
while  at  the  same  time  weighing  source  selection  heavily  in  favor  of  near 
term  development  costs.  More  trade-offs  between  RAM  and  system  performance 
need  to  be  addressed  during  the  source  selection  process.  Contractual 
incentives  should  be  included  to  stress  not  only  operational  performance, 
but  also  the  product  assurance  aspects.  While  full  compliance  may  not  be 
achievable  in  the  early  phases  of  the  program,  progressive  milestones  need 
to  be  established,  monitored,  and  related  to  incentives. 
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The  period  of  greatest  leverage  in  affecting  operational  performance, 
RAH,  and  ultimate  cost  is  the  early  design  phase.  A  great  deal  more  effort 
is  needed  "up  front"  to  prevent  problems  from  occurring.  This  involves  more 
trade-offs  during  the  conceptual  phase.  It  involves  more  emphasis  on 
simulations  of  planned  systems  and  their  interactions  with  the  expected 
tactical  environment  (and  other  related  systems).  It  means  greater  design 
emphasis,  and  early  testing,  at  the  component  and  subassembly  levels.  The 
engineering  should  stress  basics  such  as  error  budgeting,  thermal,  stress, 
and  failure  mode  analysis,  component  deratings,  parts  standardization, 
producibility ,  etc.  Modem  computer  techniques  such  as  computer-aided 
design,  finite  element  stress  and  thermal  analysis,  and  other  computer 
programs  greatly  simplify  these  engineering  studies,  and  their  use  is  strong¬ 
ly  encouraged.  Considerations  relating  to  RAM  should  be  addressed  in  the 
early  design  phase.  Significant  effort  should  be  spent  on  hardware  and 
software  simplification,  and  in  the  design  of  built  in  test  functions  that 
simplify  system  maintenance.  Carefully-prepared,  complete  documentation 
is  essential  in  early  program  phases  so  that  reference  to  trade-off  studies 
and  related  decisions  can  be  made  throughout  the  program. 

The  design  phase  of  the  program  requires  especially  careful  monitoring 
by  the  Army's  Program  Manager  and  his  technical  staff.3  Frequent  design 
reviews  should  be  held  to  assure  that  the  basic  design  concepts  are  sound, 
and  that  all  of  the  technical  issues  are  being  addressed.  The  effective¬ 
ness  of  a  program  is  largely  dependent  on  the  technical  expertise  of  the 
government  team.  It  is  the  opinion  of  the  Subgroup  that  this  technical 
expertise  has  significantly  deteriorated  over  the  last  decade,  and  this 
capability  needs  to  be  restored  if  the  Army  is  to  operate  in  a  cost  effective 
manner.  Subcontracting  the  required  evaluation  and  monitoring  efforts  to 
think-tanks  and  study  houses  is  simply  not  a  satisfactory  solution;  these 
organizations  frequently  do  not  have  direct  hardware/software  design 
experience.  As  noted  in  other  sections  of  this  report  (cf.  Sections  VI  and 
VII)  there  are  types  of  organizations  that  can  help;  but  they  cannot 
substitute  for  in-house  capability. 

An  engineering  test  program  should  be  an  integral  part  of  the  design 
cycle.  This  should  involve  program  peculiar  components,  subassemblies, 
and  subsystems  which  are  subjected  to  rigorous  performance  and  reliability 
testing  under  environmental  simulation;  special  attention  should  be  devoted 
to  software  design  and  comprehensive  software  testing  (cf.  Sections  IV  and  V 
and  Appendix  E) . 

The  testing  responsibility  at  this  point  in  the  program  should  be  the 
responsibility  of  the  system  contractor,  but  with  government  monitoring. 


3.  Relevant  "Lessons  Learned"  in  the  PATRIOT  Program  are  outlined  in 
Appendix  E,  as  discussed  by  the  Project  Manager. 
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In  this  way,  design  and  quality  deviations  will  become  apparent  to  both 
the  contractor  and  the  Army  early  in  the  program,  rather  than  at  a  much 
later  time  during  full-scale  system  testing.  In  addition,  the  component 
and  subsystem  (hardware  and  software)  testing  should  be  conducted  in  the 
contractor's  facility  when  possible.  This  allows  the  design  team  to  observe 
firsthand  any  deficiencies,  and  permits  rapid  turnaround  on  fixes.  Every 
failure  or  performance  deviation  should  be  recorded  and  analyzed,  and  fixed. 
This  approach  requires  more  up  front  funding,  and  often  a  longer  design  and 
development  cycle;  however,  the  over-all  costs  and  the  time  to  effective 
production  should  both  be  reduced. 

The  test  community  needs  to  become  involved  early  in  the  program.4  The 
Test  Integrating  Working  Group  (TIWG)  is  the  established  forum  for  this  activ¬ 
ity  and  includes  representation  from  the  program  office,  the  contractor (s) , 
the  user,  the  training  and  logistics  commands,  as  well  as  the  test  community. 
In  addition,  where  there  is  a  requirement  for  interoperability  with  other 
systems  (or  equipment),  the  TIWG  should  include  specialists  knowledgeable 
in  these  other  systems,  preferably  from  the  respective  program  offices.  Once 
established,  changes  in  the  TIWG  membership  should  be  minimal  and,  under 
normal  circumstances,  assigned  individuals  should  continue  for  the  life  of 
the  test  program.  The  chairman  of  the  TIWG  should  continue  to  be  the  Army's 
Program  Manager  or  his  deputy  rather  than  a  career-oriented  member  of  the 
test  community;  this  is  to  assure  that  the  test  program  is  compatible  with 
the  over-all  program  milestones,  that  the  test  resources  are  properly  sched¬ 
uled  and  prioritized,  and  that  the  test  program  is  adequately  funded. 

The  TIWG  should  start  Its  planning  very  early  in  the  program,  and 
all  test  plans  should  be  agreed  to  and  documented  from  the  start.  Needless 
to  say,  the  test  plan  must  be  consistent  with  the  system  specification,  and 
this  again  emphasizes  the  need  for  a  thoroughly  tailored  system  specifica¬ 
tion.  Particular  emphasis  should  be  given  to  interoperability  testing, 
which  in  the  past  has  been  often  treated  as  an  afterthought  or  sometimes 
even  ignored. 

The  management  of  the  Development  Testing  (DT)  program  should  be  the 
responsibility  of  the  Army's  Program  Manager  with  the  tests  monitored  and 
evaluated  by  the  TIWG.  Actual  testing  should  normally  be  performed  by  the 
contractor  In  his  own  facilities.  However,  it  is  recognized  that  contractors 
will  generally  not  have  specialized  facilities  such  as  flight  test  ranges, 
communications  and  jamming  test  ranges,  EMP  simulators,  etc.  These 
specialized  tests  should  be  performed  in  government  facilities  such  as  are 
available  at  Fort  Huachuca,  White  Sands  Missile  Range,  Fort  Hood,  and  others. 
However,  the  basic  management  of  the  development  testing  should  remain  with 


nt  of  an  automated 
em  loading)  is  disc 
idered;  cf.  recomme 


the  Program  Manager,  regardless  of  where  the  tests  are  performed,  with  active 
participation  of  the  contractor.  For  all  tests,  careful  attention  should  be 
given  to  Army  documentation  requirements;  tests  should  not  be  conducted  until 
requisite  documentation  is  available. 

The  primary  purpose  of  DT  is  to  assure  that  the  basic  design  meets  the 
performance  specification  under  simulated  environmental  conditions.  The 
tests  are  also  intended  to  provide  early  identification  of  areas  of  spec¬ 
ification  deviation,  so  that  the  design  can  be  modified  as  required.  In 
addition  the  inherent  system  RAM  capabilities  should  be  estimated  analyt¬ 
ically;  the  analytical  results  should  be  to  establish  goals  and  allocations 
to  subsystems  and  components.  Related  test  programs  should  be  designed  with 
successively  more  stringent  RAM  milestone  demonstrations  (starting  with  DT) 
in  order  to  assure  ultimate  compliance  for  the  production  equipment.  Early 
training  concepts  need  to  be  formulated  and  verified  during  DT  and  limited 
user  participation  is  useful  at  this  stage.  Sufficient  numbers  of  equipment 
are  needed  to  satisfy  these  needs,  and  this  must  be  recognized  by  adequate 
up  front  funding. 

Operational  Testing  (OT)  should  not  be  initiated  until  all  of  the  major 
milestones  of  DT  have  been  achieved.  The  purpose  of  operational  testing  is 
to  obtain  an  estimate  of  the  system's  over-all  operational  effectiveness  and 
suitability.  This  includes  items  such  as  survivability,  vulnerability, 
safety,  human  factors,  logistics  supportability ,  and  training  requirements. 

In  addition,  OT  objectives  include  identification  of  any  operational  defi¬ 
ciency  that  will  require  hardware  or  software  modification.  Again  it  should 
be  emphasized  that  tests  should  not  be  conducted  until  relevant  documentation 
is  available. 

Operational  tests  should  be  conducted  at  government  facilities,  under 
simulated  field  conditions,  with  progressively  greater  involvement  of  the 
user.  For  C^I  equipment,  this  testing  would  generally  occur  at  Fort 
Huachuca,  White  Sands  Missile  Range,  or  Fort  Hood  (for  ASTB) .  This  effort 
should  continue  to  be  planned,  coordinated,  and  evaluated  by  the  TIWG  group, 
and  the  over-all  test  responsibility  should  remain  with  the  Program  Manager; 
however,  the  special  requirements  of  operational  testing  must  be  clearly 
recognized  by  the  Project  Manager.  Deviations  from  those  requirements  should 
be  explicitly  justified.  Contractor  participation  at  this  stage  should  be 
limited  to  that  of  a  consultant  and  technical  adviser. 

It  should  be  recognized  that  RAM  maturation  programs  and  follow-on 
evaluations  (FOE)  will  be  needed,  since  operational  testing  (prior  to  pro¬ 
duction  decisions)  is  not  in  general  conducted  on  production  hardware  and 
software;  for  these  types  of  tests,  there  should  be  the  same  detailed 
attention  to  planning/data  collection/data  interpretation  as  that  provided 
for  prior  development  and  operational  tests. 

3 

Complex  C  I  systems  have  in  the  past  been  plagued  by  compatibility 
and  interoperability  problems  that  are  first  identified  during  operational 
testing  or  later.  As  a  result,  the  program  experiences  major  cost  overruns 


and  schedule  delays.  In  many  cases,  the  issues  wort  never  addressed  in  the 
system  specification  or  in  the  design  phase.  Obviously,  expanding  the 
operational  test  program  is  not  the  correct  approach  to  solving  this  type 
of  problem. 

To  summarize,  the  acquisition  management  approach  for  complex 
electronic  systems  should  include: 

1.  A  tailored  specification  based  on  user  needs; 

2.  A  procurement  policy  that  recognizes  total  costs  (life 
cycle  costs)  as  opposed  to  developmental  costs; 

3.  Emphasis  on  conservative  design  practices  monitored  by 
competent  government  personnel; 

4.  Early  involvement  of  the  entire  test  community  (TIWG) , 
with  adequate  attention  to  documentation  requirements; 

5.  A  test  program  that  moves  progressively  from  the  component 
and  subassembly  level  to  a  full-up  system; 

6.  A  test  program  that  is  the  responsibility  of  the  Program 
Manager,  with  explicit  attention  to  user  requirements; 

7.  Early  attention  to  reliability  and  maintenance  with 
milestone  thresholds; 

8.  Adequate  hardware  ("hangar  queens")  so  that  design 
changes  and  product  improvements  can  be  quickly 
verified; 

9.  A  follow-on  evaluation  program  to  assure  quantitative 
understanding  of  production  hardware  and  software 
designs ; 

10.  An  active  program  to  assure  quality  improvements 
throughout  production. 


IV.  SOFTWARE  DESIGN  TESTING  AND  VALIDATION:  DEVELOPMENT  OF  TEST  TOOLS 


There  is  ample  evidence  to  support  the  concern  that  there  may  be  a 
substantial  deficiency  in  the  Army's  current  ability  to  load  its  advanced 
automated  weapons  systems  with  realistic  battlefield  operating  conditions 
in  order  to  perform  thorough  and  effective  development  tests.  Proposals 
have  been  advanced  from  several  quarters  for  the  development  of  automated, 
computer-based  test  tools  to  drive  (via  simulation  and  stimulation)  the 
engineering  and  initial  production  models  of  new  automated  Army  weapons 
systems  during  DT  and  (to  somewhat  lesser  degree)  OT  (cf.  reference 
memorandum.  Appendix  F,  p.  1;  and  Section  V  of  this  report). 

The  need  for  the  proposed  automated  test  tools  is  supported  by  the  ASB. 
In  our  view  they  could  serve  in  three  important  ways  to  improve  the  Army's 
ability  to  successfully  develop  effective  automated  weapons  systems: 

1.  Validation  of  the  system's  ability  to  meet  the  stated 
requirements  (DT/OT) ; 

2.  Early  detection,  identification,  and  diagnosis  of  design 
faults  and  deficiencies  (DT  and  pre-DT  system/subsystem 
tests) ; 

3.  Source  of  guidance  and  motivation  for  system  architects  and 
designers  to  provide  for  early  consideration  of  stating 
system  requirements  in  a  "testable"  form;  encouragement  for 
supporting  system/subsystem  module  tests  concurrently  with 
very  early  system  concept  definition. 

Common  to  all  of  the  advanced  automated  systems  is  the  usage  of  em¬ 
bedded  computers  and  extensive  amounts  of  software.  Key  operational 
elements /aspects  of  system  performance  are  determined  by  the  software; 
this  fact,  which  is  central  to  how  the  embedded  computers  are  used,  implies 
that  no  phase  of  system  testing  can  be  accomplished  without  testing  the 
software  as  well  as  the  hardware.  Consequently,  test  plans  must  include 
appropriate  module,  subsystem  and  system  level  software  tests  at  all 
phases  of  system  development. 

While  it  may  seem  self-evident  that  software,  as  well  as  hardware, 
must  be  tested,  it  is  a  fact  that  the  techniques  for  reliable  and  com¬ 
prehensive  software  testing  are  entirely  different  from  comparable 
hardware  testing  techniques.  They  are  similar  in  only  the  "highest" 
philosophical  sense;  hardware  testing  techniques  serve  only  as  mildly 
useful  analogies  to  suggest  where  and  how  to  begin  the  task  of  designing 
software  tests.  The  extent  and  nature  of  these  differences  has  only 
over  the  past  five  years  or  so  begun  to  be  fully  appreciated  by  the 
systems  design  and  test  community.  As  a  result,  a  new  technology  called 
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"Software  Verification  and  Validation"  is  now  emerging  in  the  field  of 
computer  science  and  design,  consisting  of  statements  of  methodology  and 
descriptions  of  techniques;  e.g.,  Proceeding  of  Software  Verifications 
and  Validation  Symposium,  June  9-10,  1981,  MITRE;  or  NSCCA/PATE  Guidebooks 
Vol.  Ill,  June  1980,  L0GIC0N;  Tutorial:  Software  Testing  and  Validation 
Techniques ,  E.  Miller,  IEEE  Catalog  No.  EHO  138-8. 

Four  specific  recommendations  for  the  Army's  consideration  are  sub¬ 
mitted  as  a  result  of  the  above  findings.  Briefly,  they  are: 

1.  From  the  outset,  software  should  be  designed  to  be  testable; 

2.  Software  testing  should  be  made  a  part  of  DT  and  included  as 
often  as  possible  in  pre-DT  system  development  phases; 
automated  computer-based  test  tools  should  be  developed  to 
provide  appropriate  representations  of  tactical  environments 
(and  system  loading) ; 

3.  MAINSITE,  to  be  an  effective  DT  asset,  should  be  explicitly 
designed  and  equipped  for  both  software  and  hardware  testing; 

4.  To  facilitate  cost-effective  software  testing  which  will  yield 
results  which  can  be  uniformly  interpreted  and  "graded",  a 
common  library  of  software  verification  and  validation  tools 
(tests)  should  be  developed  and  used  on  an  Army-wide  basis  by 
all  of  the  developers  and  testers. 

The  following  comments  are  offered  in  support  of  these  recommendations. 
First,  the  notion  that  software  must  be  designed  to  be  testable  is  not  dis¬ 
similar  from  the  hard-learned  lesson  of  hardware  testing.  Basically,  the 
strategy  to  be  followed  by  the  software  designers  to  insure  testability 
begins  during  the  formulation  of  system  requirements  and  specifications 
with  the  designers  insisting  on  getting  an  answer  to  the  question:  "How 
can  compliance  with  this  particular  specification  be  confirmed?".  A 
simple  example  here  will  illustrate  this  strategy.  The  TRI-TAC  switch 
requirement  to  operate  satisfactorily  under  realistic  battlefield  loads 
was  not  explored  at  the  outset  for  testability,  to  the  extent  that 
"realistic  battlefield  loads"  remained  undefined  (in  a  quantitative  sense) 
until  OT  was  initiated  and  "satisfactory  operation"  was  not  ever  made 
measurable  in  terms  of  que-length  margins  and  statistics,  etc.!  Further¬ 
more,  even  if  que-length  margins  and  statistics  had  been  specified,  the 
TRI-TAC  software  was  not  designed  to  allow  these  parameters  to  be 
measured. 

In  general,  automated  systems  containing  embedded  computers  are  more 
vulnerable  to  the  failure  of  not  designing  for  testability  because  their 
software  is  much  more  intimately  responsible  for  achieving  operational 
requirements  and  specifications  than  are  current,  more  hardware-intensive 
manual  systems.  Unfortunately,  those  persons  responsible  for  early 


system  concept  formulation  usually  want,  and  often  promote,  vagueness  in 
operational  requirements  in  order  to  "retain  flexibility"  or  broaden  the 
appeal  of  the  concept.  As  a  result,  the  software  designers  are  denied 
critical  information  needed  to  insure  a  testable  design  and,  by  default, 
essential  software  performance  features  are  obscured  or  rendered  unmeasur¬ 
able/unobservable  by  well-intentioned  but  uninformed  unit  level  programmers. 
As  a  result,  intrinsic  design  flaws  in  the  software  remain  unexposed  until 
OT  or  even  later  phases  of  system  design.  Careful  adherance  to  the  tightly 
disciplined  early  "design  for  testing"  methodology  now  being  articulated  in 
the  literature  is  an  essential  step  in  remedying  current  deficiencies  in 
the  Army's  testing  of  automated  systems. 

The  second  recommendation  to  include  software  testing  as  a  part  of  DT 
can,  of  course,  be  fully  implemented  only  if  the  first  recommendation  has 
been  implemented.  The  important  point  here  is  that  DT  is  the  appropriate 
phase  of  system  development  for  confirming  proper  software  performance, 
including  acceptable  software  unit  and  subsystem  tests.  The  intimate 
involvement  of  the  software  with  operational  aspects  of  the  system  invites 
confusion  in  the  Army  development/testing  process  and  has  often  led  to 
deferral  of  comprehensive  software  testing  to  the  OT  or  later  phase 
(e.g.,  PATRIOT).  Early  software  testing  allows  correction  of  design  flaws 
at  much  less  expense  than  correction  later  and,  furthermore,  often  allows 
a  technically  superior  solution  involving  architectural  and/or  hardware 
changes  which  may  become  impractical  later  in  the  development  cycle. 

The  third  recommendation  regarding  MAINSITE  is  recognition  of  the 
intended  centrality  of  the  role  MAINSITE  is  to  play  in  DT  for  automated 
tactical  systems.  To  date,  most  of  the  planning  for  the  MAINSITE  capabil¬ 
ities  appear  to  have  dealt  with  its  abilities  for  testing  and  demonstrating 
hardware  features  of  the  system  under  test.  While  some  form  of  software 
testing  is  currently  expected  to  be  executed  by  the  system,  implementing 
the  previous  two  recommendations  will  enable  the  MAINSITE  planners  and 
designers  to  fully  embrace  the  potential  for  testing  the  software  as  well 
as  the  hardware  in  DT.  A  broad  collection  of  software  verification  and 
validation  tools  can  and  should  be  implemented  and  maintained  up-to-date 
in  MAINSITE  facilities.  Their  availability  at  MAINSITE  would  also  serve 
the  purpose  of  reinforcing  and  guiding  the  particulars  of  how  to  "design 
for  test"  prior  to  DT;  i.e.  MAINSITE  would  serve  to  remind  and  suggest  to 
the  system  designers  what  should  be  anticipated  in  DT.  By  virtue  of  the 
uniqueness  of  MAINSITE,  software  verification  and  validation  tools  will 
tend  naturally  to  be  standardized  among  Army  developers.  At  a  minimum, 
equipping  and  tasking  MAINSITE  with  software  testing  will  tend  to  standardize 
software  interfaces  critical  for  testing  as  well  as  to  standardize  many  of 
the  "measures"  of  software  performance  and  design  validity. 

Finally,  the  fourth  recommendation  to  develop  a  common  library  of  soft¬ 
ware  verification  and  validation  tools  should  greatly  relieve  the  cost  burden 
of  software  testing  on  developers  as  well  as  aid  the  testers  in  becoming  a 
more  constructive  and  better  understood  player  on  the  Army  system  development 
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team.  Nearly  all  of  the  current  Army  systems  under  development  with 
embedded  computers  can  exhibit  a  discouraging  and  lengthy  list  of  sample 
incidents  where  the  test  and  testers  from  one  phase  of  the  development 
are  at  odds  with  those  of  a  later  phase.  It  is  difficult  to  separate 
design  problems  from  communication/interpretation  problems  in  these 
incidents.  Clearly,  employment  of  methodology  —  a  common  library  of 
tools  —  would  be  a  major  step  toward  more  effective  orderly,  development 
and  testing. 

The  newness  of  the  emerging  software  verification  and  validation  con¬ 
cepts  as  a  discipline  places  a  strong  technical  obligation  on  the  Army  and 
carries  all  the  usual  risks  of  a  developing  technology.  However,  the 
newness  also  presents  the  Army  with  an  opportunity  to  assume  and  provide 
leadership  in  a  vital  —  perhaps  critical  —  new  discipline  with  applica¬ 
tion  in  the  commercial  and  industrial  as  well  as  military  sectors. 

In  addition  to  the  preceding  recommendations,  it  is  also  strongly 
suggested  that  consideration  be  given  to  the  suggestion  made  in  the  refer¬ 
ence  memorandum  reprinted  in  Appendix  F,  p.  4.  It  is  proposed  that  the 
previously-discussed  automated  test  systems  (to  provide  the  test  environment 
system  loading)  be  developed  by  contractors  other  than  the  system  development 
contractors  —  in  parallel  with  the  development  of  the  basic  system.  It  is 
recognized  that  additional  coordination  would  be  required  during  engineering 
development;  however,  the  advantages  (as  stated  in  the  reference)  could  be 
highly  significant  in  terms  of  program  time  scales,  cost  and  performance. 
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V.  LESSONS  LEARNED  FROM  US  AIR  FORCE  SOFTWARE  DEVELOPMENT; 


RELATIONSHIP  TO  US  ARMY  PLANS  FOR  TESTING 


It  has  been  reported  5  that  a  survey  of  discrepancy  reports  of 
eighteen  US  Air  Force  (A/F)  projects  that  were  subjected  to  detailed 
review  (i.e.,  verification  and  validation)  processes  indicated  that  cer¬ 
tain  types  of  software  development  problems  were  encountered  repeatedly. 
Although  the  programs  varied  in  application,  language,  development  method, 
size  and  complexity,  certain  types  of  software  problems  occurred  repeatedly. 
The  A/F  found  that  an  understanding  of  common  problem  areas  is  an  invaluable 
aid  to  the  software  evaluator  and  that  such  knowledge  is  transferrable  from 
project  to  project. 

The  predictable  areas  of  difficulty  were  found  to  be: 

REQUIREMENTS  PROBLEMS 
o  Incomplete  requirements 
o  Inconsistent  requirements 
o  Incorrect  requirements 

o  Untestable ,  ambiguous,  and  questionable  requirements 

DESIGN  AND  CODE  PROBLEMS 


o 

Initialization,  reinitialization, 

restarts 

o 

Flags,  counters,  indices 

o 

Data  definition  and  usage 

o 

Mathematics 

0 

Timing,  interruptibility ,  process 

allocation 

o 

Interfaces 

o 

Miscellaneous  errors 

o 

Questionable  design  and  poor  programming  practices 

DOCUMENTATION  PROBLEMS 

o 

"As-Built"  specifications 

o 

User  documentation 

5.  LOGICON  Report  R: SED-80204-III ,  Prepared  for  BMO/MNNC,  Norton  Air  Force 
Base,  California,  dated  June  1980,  "NSCCA/PATE  Guidebooks,  Volume  III: 
Lessons  Learned  from  Past  NSCCA/PATE  Efforts".  Related  discussion 
conducted  at  LOGICON,  San  Pedro,  California,  on  21  April  1981,  with 
member  of  Subgroup  on  Testing  of  Electronic  Systems. 


The  A/F  software  applications  included  programs  relating  to  command  and 
control,  chart-generation,  targeting,  data  base  generation,  and  range 
safety.  Some  were  in  real  time,  some  were  written  in  an  assembly  language, 
some  were  written  in  Fortran,  some  were  hosted  by  UYK  series  processors  and 
others  were  hosted  by  such  commercial  processors  as  the  IBM  360  and  CDC  3300. 

The  diversity  of  the  applications  leaves  little  doubt  that  the  indi¬ 
cated  problem  areas  will  be  representative  of  those  that  will  be  found  in 
the  development  of  software  for  the  Army.  At  this  point  in  time  we  know 
that  software  problem  areas  are  among  the  items  that  should  be  explicitly 
tested  and  evaluated  prior  to  the  DSARC  I,  II  and  III  decision  points; 
therefore  it  is  useful  to  examine  the  Army's  current  and  planned  T&E 
capabilities  to  find,  evaluate,  and  fix  the  kinds  of  problems  that  are 
expected  to  be  encountered  in  the  software  items  utilized  in  sophisticated 
systems . 

In  that  context,  briefings  from  the  MI COM  System  Software  Center6 
indicated  their  sensitivity  to  requirements  and  development  problems. 

With  the  definition  of  software  testing  presented  in  Figure  1,  a  clear 
understanding  of  cost  problems  was  demonstrated  in  the  discussion  of  Figure 
2,  where  the  cost  of  errors  was  correlated  with  the  stage  of  development  in 
which  the  errors  were  detected.  Figure  3  was  used  to  relate  software  test¬ 
ing  to  a  pyramid,  with  requirements  analysis  and  software  design  analysis 
as  elements  of  the  foundation,  and  integrated  program  testing  at  the  tip. 

In  reference  to  the  analogy  of  the  pyramid  —  one  major  concern  of  the 
Subgroup  is  that,  for  sophisticated  systems  with  embedded  computers,  the 
current  approach  seems  to  involve  the  expenditure  of  large  amounts  of  money 
and  effort  for  testing  at  the  tip,  with  far  less  attention  to  testing  at 
the  foundation.  Furthermore,  there  seem  to  be  "disconnects"  between 
apparent ly-suit able  plans/approaches  as  discussed  by  software-oriented 
organizations  within  the  Army  (e.g.,  the  MICOM  Missile  System  Software 
Center,  or  the  BMDSCOM  Testing  Organization)  and  current  implementation 
in  tactical  projects. 

Similarly,  there  would  appear  to  be  "disconnects"  relative  to  current 
planning  of  two  major  T&E  test  systems,  MAINSITE7  (Ft.  Huachuca)  and  ATSTB8 
(Ft.  Hood);  essential  aspects  of  the  facilities  are  shown  in  Figures  4  and  5. 
As  also  discussed  in  Section  IV,  the  test  resources  seem  to  be  directed 
primarily  toward  hardware  problems,  with  insufficient  planning  for  software 
testing.  For  MAINSITE,  a  module-by-module  comparison  has  been  made  (by  the 


6.  Presentation  on  MICOM  System  Testing  Policy,  13  April  1981, 

Redstone  Arsenal  (cf.  Appendix  B,  p.  B-7). 

7.  Modular  Automated  Integrated  Systems  Interoperability  Test  and 
Evaluation  System. 

8.  Automated  Tactical  Systems  Test  Bed 
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Subgroup)  with  the  types  of  problems  evidenced  by  the  A/F  analyses;  although 
the  analysis  has  been  cursory,  and  may  be  based  on  an  inadequate  understand¬ 
ing  of  recent  plans,  it  would  seem  that  relatively  few  of  the  common  problem 
areas  identified  by  the  A/F  would  be  adequately  investigated  during  DT  at 
MAINSITE  or  OT  at  ATSTB.  Nor  does  it  seem  that  the  experience/understanding 
of  software-oriented  Army  organizations  has  been  appropriately  exploited. 

It  is  important  that  the  foregoing  remarks  not  be  regarded  as  an  indict¬ 
ment  of  Army  test  and  evaluation  facilities  in  general,  or  of  other  aspects 
of  MAINSITE  and  ATSTB  planning.  In  fact,  the  Subgroup  has  been  very  favorably 
impressed  by  the  general  excellence  of  facilities  —  existent  and  planned  — 
and  by  the  competence  and  dedication  of  the  associated  technical  staffs. 

The  problems  relate  to  the  evident  fact  that  plans  for  future  T&E 
facilities  seem  to  be  based  primarily  on  hardware  test  experience  —  experi¬ 
ence  that  bears  little  relationship  to  software  testing  needs.  Additional 
coordination  within  the  Army,  with  consideration  of  types  of  tests  to  be 
conducted  at  the  various  facilities  —  and  the  consequent  requirements  for 
test  capabilities  —  would  appear  to  be  urgent;  in  particular,  the  require¬ 
ments  for  software  testing  should  be  reemphasized  in  all  planning. 
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Figure  1 

SOFTWARE  TESTING 
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VI.  NEED  FOR  IMPROVED  EVALUATION  OF  DESIGN  CONCEPTS  AND  EARLY 
TEST  DESIGN;  RELATIONSHIP  TO  IN-HOUSE  CAPABILITIES 


A.  Introduction;  Need  for  Improved  Evaluation  of  Design  Concepts 

The  design  problems  that  have  been  revealed  in  many  systems  that  are 
well  along  in  the  development  and  testing  cycle  are  clear  evidence  of  lack 
of  competent  attention  to  early  design  concepts.  Such  failure"  must  surely 
reflect  the  erosion  of  quality  of  in-house  capability  and  the  need  for 
improvement . 

3 

As  C  I  and  other  computer-based  systems  become  more  and  more  complex, 
the  competency  and  thoroughness  with  which  the  original  system  design  con¬ 
cept  is  developed  become  of  overriding  importance.  Whether  this  concept  is 
created  by  contractor  or  in-house  engineers  is  immaterial.  In  either  case, 
in-house  personnel  must  not  only  have  a  good  understanding  of  the  design 
concept,  but  must  be  able  to  relate  it  to  the  systems  with  which  it  must 
interact,  both  available  and  yet  to  come. 

It  is  recognized  that  many  personnel  actions  in  recent  years  —  some 
not  under  the  control  of  the  Army  —  have  had  a  severe  impact  on  the  tech¬ 
nical  capabilities  of  the  Army  in-house  R&D  structure.  Arbitrary  cuts  in 
manpower,  arbitrary  limitations  on  laboratory  average  grades,  and  arbitrary 
ceilings  on  the  salaries  of  the  higher  grades  must  ultimately  result  in 
erosion  of  the  quality  and  effectiveness  of  personnel  whose  responsibilities 
are  those  of  the  "wise  buyers"  needed  for  effective  governmental  control. 

Major  efforts  should  be  made  to  improve  the  Army's  capabilities  for 
detailed  evaluation  of  design  concepts.  If  possible,  the  related  personnel 
actions  should  be  reversed;  in  any  event,  careful  attention  should  be  given 
to  organizing  the  optimal  use  of  available  talent.  Additional,  further 
consideration  should  be  given  to  the  acquisition  of  contractual  support  on 
a  long-term,  continuing  basis  to  assist  in  this  area. 

B.  Need  for  Early  Planning  of  Development  and  Operational  Tests 

Briefers  from  the  Testing  and  Evaluation  community  have  presented  the 
Ad  Hoc  Working  Group  with  their  views  that  a  considerable  need  exists  for 
more  sophisticated  facilities  than  those  presently  available.  There  has 
not  been  an  equivalent  emphasis  on  early  clarification  of  concepts,  and  on 
early  planning  for  development  and  operational  tests.  The  PATRIOT  experience 
(cf.  Appendix  E)  indicates  a  need  for  the  early  integration  of  test  community 
requirements  into  the  over-all  design  of  the  development,  testing,  and 
evaluation  of  complex  systems,  as  well  as  a  need  for  additional  attention  to 
software  testing  and  to  software /hardware  integration.  The  Subgroup  believes 
that  earlier  and  greater  emphasis  on  both  general  system  and  test  design 
might  have  reduced  the  downstream  problems  that  have  caused  expensive  delays 
in  the  Army's  ability  to  field  complex  systems.  Costs  escalate  in  a  non¬ 
linear  fashion  when  problems  are  not  intercepted  in  advance  of  operational 
testing. 


The  Army  lacks  the  equivalent  of  an  Aerospace  Corporation  or  a  JHU/APL 
to  assist  in  concept  formulation/evaluation  and  in  the  definition/monitoring 
of  testing.  This  being  so,  it  would  seem  reasonable  to  look  to  the  Army 
Science  Laboratory  research  and  development  community  for  technical  advice. 
However,  such  help  is  not  forthcoming,  or  tends  to  be  available  in  only  a 
limited  way.  In  the  first  place,  considerable  distances  separate  these 
laboratories  from  the  proving  grounds,  i.e.,  New  Jersey  to  Arizona.  In  ad¬ 
dition,  it  may  be  said  that  some  of  the  Army  Science  Laboratories  are  in 
need  of  technical  advisers  themselves,  in  order  to  carry  out  significant 
aspects  of  their  missions.  There  has  been  a  long  process  of  erosion  in  the 
quality  of  the  various  laboratories  and  a  decrease  in  the  number  of  talented 
personnel  available  to  them.  This  is  attributed  to  salary  and  GS  level 
limitations  that  are  not  competitive  with  industry  and  therefore  severely 
limit  the  ability  of  the  laboratories  to  attract  and  retain  qualified 
engineers  and  scientists. 

Given  all  of  the  above,  together  with  the  fact  that  the  testing  and 
evaluation  missions  are  so  closely  defined  as  to  virtually  preclude  the 
hiring  of  innovative  engineers  and  scientists,  it  is  understandable  that 
inadequate  attention  has  been  given  to  initial  project  planning  and  design. 

It  is  also  plausible  to  question  the  present  wisdom  of  the  historical 
separation  of  the  research  and  development  activities  from  the  testing  and 
evaluation  facilities  on  a  geographical  basis.  Once  considered  remote  and 
undesirable  locations,  Sunbelt  areas  have  experienced  an  influx  of  population 
in  recent  years,  and  excellent  young  engineers  and  scientists  are  now  being 
graduated  from  university  programs  in  these  states.  This  suggests  that  it 
might  now  be  quite  realistic  to  expect  that  a  significant  percentage  of  the 
available  talent  would  be  interested  in  employment  in  Southwest  locations, 
if  there  were  also  a  professionally  attractive  mix  of  research,  development 
and  consulting  work  to  do  there.  There  would  then  be  the  additional  possi¬ 
bility  that  technical  advice  could  be  available  to  the  testing  and  evaluation 
projects  at  such  sites  as  Fort  Huachuca,  White  Sands  and  Dugway.  This  sug¬ 
gestion  would  not  resolve  the  salary  and  promotion  problems  that  burden  the 
federal  employment  structure,  but  it  does  hold  promise  regarding  the 
possibility  of  an  influx  of  new  talent  for  the  Army's  over-all  effort,  even 
if  much  of  it  departed  when  further  promotions  and  raises  were  denied  by 
government  regulations. 

It  is  also  worth  pointing  out  that  some  of  the  software  problems  that 
have  plagued  the  Army's  complex  systems  in  the  past  might  be  alleviated  in 
the  future  if  young  engineers  and  scientists  could  be  attracted  to  T&E 
locations  and  used  as  advisers  in  the  early  phases  of  test  design.  Use  of 
the  computer  is  now  an  integral  part  of  engineering  and  scientific  education. 
It  is  taken  for  granted,  rather  than  being  viewed  as  something  to  be  added 
on  (as  it  was  added  on  to  the  set  of  professional  skills  of  older  workers 
who  have  carried  the  project  responsibilities  for  the  last  20  years).  If 
this  newer  point  of  view  could  be  brought  into  all  aspects  of  technical 
activity,  particularly  into  the  initial  phases  of  system  and  test  design, 
it  could  save  the  Army  considerable  time  and  expense. 
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VII.  NEED  FOR  TECHNICAL  CONTINUITY  THROUGHOUT  DESIGN  AND  TESTING; 
RELATIONSHIP  TO  US  ARMY  ORGANIZATION  FOR  TESTING 


As  a  system  evolves  from  concept  (TRADOC) ,  through  demonstration 
(contractor,  development  laboratory)  and  development  (contractor,  Project 
Manager),  to  fielding  (readiness  side  of  commodity  command),  technical 
continuity  (on  the  part  of  the  government)  tends  to  exist  only  in  an 
archival  sense.  In  rare  instances,  not  due  to  systematic  process,  individ¬ 
uals  may  shift  jobs  to  follow  a  system  through  this  cycle,  but  inadequate 
records  (rationale  for  past  choices,  data)  and  insufficient  personal 
recollections  tend  to  dominate  this  problem.  For  high  technology,  complex 
systems,  this  situation  is  exacerbated  by  longer  acquisition  cycles. 

Transfer  of  organizational  responsibility  at  system  milestones  contri¬ 
butes  to  the  indicated  lack  of  technical  continuity.  Even  within  the  time 
span  a  system  remains  under  the  responsibility  of  one  office  (e.g.,  PM), 
however,  rotation  of  technical  personnel,  emphasis  on  schedules  and  costs, 
and  tendencies  toward  insufficient  documentation  lead  to  erosion  of 
technical  knowledge. 

As  discussed  in  Section  VI,  the  Army  should  make  every  effort  to  im¬ 
prove  in-house  capabilities;  and  it  may  be  desirable  more  systematically 
to  transfer  managers  and  engineers  as  system  responsibility  is  shifted. 
Additionally,  however,  the  Army  could  appropriately  consider  establishing  an 
external  technical  monitor  (e.g.,  an  FCRC  or  hardware-oriented  university 
laboratory)  tasked  to  maintain  technical  continuity  for  major  systems. 

The  lack  of  over-all  program  technical  continuity  includes  a  sub-set 
of  problems  associated  with  testing,  particularly  for  complex  electronic 
systems.  These  relate  to  definition  of  test  needs  and  organizational 
responsibilities . 

As  discussed  in  previous  Army  Science  Board  reports9,  test  data  to  be 
gathered  by  the  contractor,  in  DT,  and  in  OT  —  over  the  pre-DSARC  III  life 
of  a  system  —  need  to  be  coordinated  early.  Particularly  for  electronic 
systems,  key  data  and  results  need  to  be  better  related  in  an  auditable  man¬ 
ner  to  critical  requirements  in  the  Letter  of  Agreement ,  Decision  Coordinat¬ 
ing  Paper,  and  Required  Operational  Capability  documentation;  rationale 
underlying  the  definition  of  pertinent  test  data  is  generally  insufficient. 
Similarly,  a  holistic  approach  to  assigning  tests  to  be  done  at  each  stage 
(concept,  DT,  OT,  etc.)  could  come  closer  to  ensuring  complete  test  coverage 
of  truly  key  items  without  redundancy.  For  sophisticated  electronic  systems 
examined  by  the  Subgroup,  it  may  be  more  difficult  (but  not  less  important) 
to  identify  specific  test  procedures  and  measurements  to  be  made  in  such  a 
life-cycle  manner;  however,  intensive  front-end  analysis  containing  a  logical 


9.  1980  Summer  Study  on  Statistical  Techniques  in  Testing,  7-11  July  1980, 

pp.  6-7;  Report  of  the  Panel  on  Design  of  Army  Tests,  1  May  1981,  pp.  2-3. 
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definition  of  system  criteria,  and  related  tests  to  measure  satisfaction 
of  those  criteria  over  time,  could  improve  the  value  of  testing. 

The  Army  should  strengthen  and  enforce  policies  aimed  at  early, 
intensive,  logical,  and  coordinated  determination  of  the  entire  testing 
need  for  each  system. 

All  of  the  problems  relating  to  technical  testing  continuity  are  com¬ 
plicated  by  the  cumbersome,  fragmented  organizational  structure  of  the  Army 
test  community.  The  assignment  of  responsibility  for  the  various  facets  of 
test  policy,  management,  accomplishment,  and  presentation  of  results  seems 
inconsistent  and  inefficient.  The  members  of  this  Panel  had  difficulty  in 
reconciling,  or  in  some  cases  understanding,  the  organizational  responsibil¬ 
ities.  For  example: 

1.  OTEA  is  the  unbiased,  high  level  agency  conforming  to  the 
independent  test  evaluator/manager  philosophy,  but  other  agencies  have 
the  oversight  role  in  certain  tests. 

2.  Tests  appear  to  be  assigned  to  test  activities  within  a  major  command 
with  (in  some  cases)  a  sense  of  arbitrariness;  specialization  of  skills  and 
test  equipment  in  the  several  activities  would  suggest  that  all  systems  of  a 
type  be  under  the  test  jurisdiction  of  a  commodity-oriented  activity. 

It  would  appear  that  greater  objectivity,  continuity,  coordination,  and 
efficiency  would  result  from  concentrating  testing  responsibility,  in  contrast 
to  the  current  structure.  No  specific  recommendation  is  offered  in  this  area, 
but  rethinking  of  the  relationships  of  the  developer  (DARCOM) ,  the  requirements 
generator  (TRADOC) ,  both  having  considerable  internal  test  and  evaluation 
assets,  and  the  independent  agencies  (OTEA)  might  clarify  a  currently  confusing 
organizational  alignment;  as  an  interim  measure  it  seems  possible  that  an 
executive  steering  group  (composed  of  DARCOM,  TRADOC  and  OTEA  representatives) 
could  provide  useful  coordination  and  guidance.  The  plethora  of  test  boards, 
proving  grounds,  ranges,  and  test  beds  under  various  headquarters  may  contrib¬ 
ute  to  unnecessary  duplication,  e.g.,  the  question  of  complementarity  for 
MAINSITE  and  ATSTB  and  their  relationship  to  similar  data  measuring  systems 
in  place  or  planned  at  CDEC10  and  NTC11. 

In  conclusion,  it  appears  that  the  Army  should  reassess  its  organization 
for  testing,  with  a  view  toward  increasing  efficiency  through  centralization. 
The  use  of  an  external  technical  test  contractor  to  provide  testing  continuity 
might  be  considered.  Additionally,  a  separate  agency/contractor  might  fulfill 
the  independent  validation,  verification,  and  evaluation  role.  Expensive, 
potentially  competing  test  and  evaluation  systems  (e.g.,  MAINSITE,  ATSTB,  CDEC, 
NTC)  need  to  be  examined  to  ensure  complementarity  and  to  minimize  duplication. 


10.  Combat  Developments  Experimentation  Command 

11.  National  Training  Center 
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DEPARTMENT  OF  THE  ARMY 
ARMY  SCIENCE  BOARD 
OFFICE  OF  THE  ASSISTANT  SECRETARY 
WASHINGTON,  D.C.  20310 


1  9  CEO  T980 


Mr.  Alvin  R.  Eaton 
Assistant  Director 

Supervisor,  Fleet  Systems  Department 
The  Johns  Hopkins  University 
Applied  Physics  Laboratory 
Johns  Hopkins  Road 
Laurel,  MD  20810 

Dear  Mr .  Eaton, 


I  would  appreciate  your  chairing  an  Army  Science  Board  Panel  to 
assess  the  testing  of  sophisticated  electronic-intensive  Army 
systems  as  requested  in  the  enclosed  letter.  A  list  of  potential 
participants  is  also  enclosed. 

The  increasingly  complex  problem  of  testing  new  systems  involves 
high  costs  of  extensive  testing  versus  the  risk  inherent  in  more 
economical,  but  less  stressing,  tests.  The  Panel  should  address 
this  issue  both  in  broad  terms  (Army  concepts  and  plans)  and  in 
relation  to  specific  facilities.  As  usual,  the  Army  Science 
Board  members  participating  in  the  study  are  responsible  for  the 
conclusions  and  recommendations  in  the  final  report. 

I  look  forward  to  hearing  of  your  progress  in  this  area  at  the 
Spring  General  Membership  Meeting  in  March. 


Sincerely , 


9-. 

^  J.  Ernest  Wilkins,  Jr. 
Cha irman 


2  Inclosures 
As  stated 
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PARTICIPANTS 


ARMY  SCIENCE  BOARD  AD  HOC  SUB-GROUP 
ON 

TESTING  OF  ELECTRONIC  SYSTEMS 


MR.  ALVIN  R.  EATON,  CHAIRMAN 
ASSISTANT  DIRECTOR 

SUPERVISOR,  FLEET  SYSTEMS  DEPARTMENT 

THE  JOHNS  HOPKINS  UNIVERSITY 

APPLIED  PHYSICS  LABORATORY 

JOHNS  HOPKINS  ROAD 

LAUREL,  MD  20810 

(301)  953-7100  X558 


LTG  AUSTIN  W.  BETTS  (USA-RET) 
SENIOR  VICE  PRESIDENT  FOR 
OPERATIONS 

SOUTHWEST  RESEARCH  INSTITUTE 
POST  OFFICE  DRAWER  28510 
SAN  ANTONIO,  TX  78284 
(512)  684-5111  X2202 

DR.  E.  O.  HARTIG 
VICE  PRESIDENT 
RESEARCH  AND  ENGINEERING 
GOODYEAR  AEROSPACE  CORPORATION 
1210  MASSILLON  ROAD 
AKRON,  OH  44315 
(216)  794-7266 

DR.  GEORGE  H.  HEILMEIER 
VICE  PRESIDENT 
CORPORATE  RESEARCH,  DEVELOP¬ 
MENT  AND  ENGINEERING 
TEXAS  INSTRUMENTS,  INCORPORATED 
POST  OFFICE  BOX  225474,  MS  400 
DALLAS,  TX  75265 
(214)  995-5975 

DR.  L.  WARREN  MORRISON 
PRESIDENT 

DIRECT  DATA  CORPORATION 
3201  N.  ALAMEDA  STREET 
COMPTON,  CA  90222 
(213)  637-0701 


DR.  IRENE  C.  PEDEN 
PROFESSOR  OF  ELECTRICAL 
ENGINEERING 

UNIVERSITY  OF  WASHINGTON 
SEATTLE,  WA  98195 
(206)  543-8025/2150 

MR.  JUAN  SANDOVAL 
VICE  PRESIDENT  AND  DIRECTOR 
OF  ENGINEERING 
AEROJET  ELECTRO  SYSTEMS 
COMPANY 

1100  W.  HOLLYVALE  STREET 
AZUSA,  CA  91702 
(213)  334-6211  X4214 

DR.  JOHN  R.  TOOLEY 
DEAN  OF  ENGINEERING 
UNIVERSITY  OF  EVANSVILLE 
POST  OFFICE  BOX  329 
EVANSVILLE,  IN  47702 
(812)  479-2651 

DR.  ANDREW  J.  VITERBI 
EXECUTIVE  VICE  PRESIDENT 
LINKABIT  CORPORATION 
10453  ROSELLE  STREET 
SAN  DIEGO,  CA  92121 
(714)  457-2340  X616 
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DEPARTMENT  OF  THE  ARMY 
OFFICE  OF  THE  ASSISTANT  SECRETARY 

WASHINGTON,  D.C.  20310 


FEPt-Y  TO 
ATTENTION  OF 


Dr.  J.  Ernest  Wilkins,  Jr. 
Deputy  General  Manager 
EG&G  Idaho,  Incorporated 
Post  Office  Box  1625 
Idaho  Falls,  Idaho  83401 


Dear  Dr.  Wilkins: 

It  is  requested  that  you  appoint  a  Panel  of  approximately  eight  Army 
Science  Board  members  to  examine  the  Testing  of  Electronic  Systems. 

As  increasingly  sophisticated  Army  systems  are  developed,  the  testing 
and  evaluation  of  these  systems  during  the  acquisition  cycle  become 
more  challenging.  Particularly  difficult  is  the  testing  of  the  C1 2 3i 

and  computer-based  portions  of  these  systems  in  realistic  battlefield 
environments  that  include  anticipated  levels  of  Input-output,  system 
software  loading,  electronic  threats,  and  maintenance.  Present  ap¬ 
proaches  include  simulation  and  field  exercise.  The  expense  of  elab¬ 
orate  testing  must  be  weighed  against  the  risk  of  detecting  potential 
system  failure  mechanisms  and  operational  difficulties  only  after 
development  and  fielding. 

The  ASB  Panel  should  examine  the  overall  facets  of  this  subject,  speci¬ 
fically  addressing  the  following: 

1.  Ar^  Army  concepts,  plans  and  equipments  adequate  for  the  testing 
of  modern  CJI  and  computer-based  systems? 

2.  What  changes  should  be  made,  if  any? 

This  investigation  should  include  an  assessment  of  relevant  testing 
facilities,  including  TRI-TAC's  Joint  Test  Facility  at  Fort  Huachuca, 
the  Automated  Systems  Test  Bed  at  Fort  Hood,  and  plans  for  the  Modular 
Automated  Integrated  Systems  Interoperability  Test  and  Evaluation 
(MAINSITE)  System.  The  adequacy  of  facilities,  test  equipment,  pro¬ 
cedures  and  plans  to  support  testing  of  ASAS,  PLRS,  and  TACFIRE  should 
be  addressed.  In  addition,  suggestions  for  satisfactorily  operation¬ 
ally  testing  future  software  systems  would  be  appreciated. 


SUMMARY  OF  AO  HOC  SUBGROUP  MEETINGS 
INDEX  OF  PRESENTATIONS 


Appendix  B 


SUMMARY  OF  AD  HOC  SUBGROUP  MEETINGS 


The  Ad  Hoc  Subgroup  on  Testing  of  Electronic  Systems  held  six  meetings 
during  the  study.  Meeting  topics  covered  briefings  on  testing  procedures 
and  policies  and  also  visits  to  a  number  of  facilities.  Summaries  of  the 
meetings  follow: 

29-30  January  Meeting  held  at  the  Applied  Physics  Laboratory,  The 
Johns  Hopkins  University,  Laurel,  Maryland:  The  primary  objective  of  the 
meeting  was  to  orient  ASB  members  on  how  testing  of  electronic  systems  is 
currently  conducted  within  the  Army  and  to  inform  the  members  of  the  test 
facilities/sites  that  are  utilized  for  this  testing.  During  this  meeting, 
the  OASA(RDA)  outlined  the  tasks  to  be  addressed  by  the  Ad  Hoc  Subgroup. 
ODCSRDA  presented  an  overview  of  the  Army  testing  and  DARCOM  addressed 
how  the  Army  currently  conducts  development  testing  on  electronic  equip¬ 
ment  plus  the  test  facilities/sites  utilized  for  testing  this  equipment 
and  the  associated  software.  TRADOC  briefed  on  each  of  their  test 
facilities/sites  that  are  utilized  for  the  operational  testing  of  electronic 
systems  and  OTEA  addressed  how  the  Army  currently  conducts  operational  tests 
on  electronic  systems  and  how  the  requirements  for  test  sites,  equipment, 
and  instrumentation  for  an  operational  test  are  established. 

3-4  March  meeting  held  at  the  Applied  Physics  Laboratory,  The  Johns 


Hopkins  University,  Laurel,  Maryland:  The  primary  objectives  of  the  meet¬ 
ing  were  to  discuss  the  need  for  the  Automated  Tactical  Systems  Test  Bed 
plus  obtain  a  perspective  of  similar  testing  in  the  Navy  and  the  views  of 
other  agencies /activities  not  directly  associated  with  testing  in  the  Army. 
The  rationale  for  developing  the  Automated  Tactical  Systems  Test  Bed  was 
presented  to  the  Subgroup.  The  Subgroup  was  briefed  on  what  the  testing 
community  is  expected  to  present  to  the  ASARC  from  the  ASARC  perspective. 
Also  briefed  was  the  OSD  perspective  of  Army  testing  and  Navy  testing  of 
communications  and  C^I  developmental  test  and  evaluation. 

18  March  meeting  held  at  the  TRADOC  Combined  Arms  Test  Activity, 

Fort  Hood,  Texas:  The  primary  objective  of  the  meeting  was  to  orient  the 
ASB  members  on  the  testing  capabilities  and  facilities  of  the  TRADOC  Com¬ 
bined  Arms  Test  Activity  (TCATA) .  During  this  meeting,  the  ASB  members 
were  briefed  on  TCATA' s  force  development  test  and  experimentation  testing 
mission,  the  testing  of  electronic  systems,  the  design  of  tests,  TRADOC' s 
test  methodology,  and  instrumentation  to  include  the  Automated  Tactical 
Systems  Test  Bed  and  the  Mobile  Automated  Field  Instrumentation  System. 

The  members  also  visited  TCATA's  ADP  facilities,  observed  some  of  their 
instrumentation,  received  a  briefing  on  ongoing  Ml  Tank  testing,  and 
inspected  the  Ml  Tank. 


13-14  April  meeting  held  at  the  US  Army  Missile  Command,  Redstone 
Arsenal,  Alabama:  The  primary  objective  of  the  meeting  was  to  gain  an 
understanding  of  missile  and  associated  software  systems  testing.  Dis¬ 
cussions  were  held  on  the  testing  of  PATRIOT,  HAWK,  PERSHING,  and  Air 
Defense  Command  and  Control.  Additional  discussions  included  MI COM’s 
system  testing  policy,  software  testing  policy  and  testing  after  oper¬ 
ational  test  III.  The  Subgroup  also  toured  many  of  the  facilities  at 
Redstone  Arsenal  to  include  a  briefing  on  testing  by  BMDSCOM. 

19-21  May  meeting  held  at  White  Sands  Missile  Range,  White  Sands, 
New  Mexico  and  the  US  Army  Electronic  Proving  Ground,  Fort  Huachuca, 
Arizona:  The  primary  objective  of  this  meeting  was  an  orientation/ 
introduction  to  Army  test  facilities.  Items  observed/discussed  while  at 
WSMR  were:  MLRS  testing  plus  observing  a  test  firing,  software  testing, 
range  control,  drone  control  center,  and  electronic  countermeasure  test¬ 
ing.  Items  observed/discussed  while  at  Fort  Huachuca  were:  update  of 
MAINSITE,  plans  for  testing  PLRS  plus  a  test  site  visit,  software  test¬ 
ing,  TRI-TAC  test  facility,  and  countermeasures  testing.  The  Subgroup 
also  visited  the  Intelligence  Security  Board  at  Fort  Huachuca  and  the 
Electromagnetic  Environmental  Test  Facility  at  Tucson,  Arizona. 


9-10  July  meeting  held  at  the  Applied  Physics  Laboratory,  The  Johns 
Hopkins  University,  Laurel,  Maryland:  The  primary  objectives  for  the 
meeting  were  to  gain  insight  into  how  the  TECOM  test  facilities  are 
utilized  and  to  discuss  the  production  of  a  final  report.  TECOM  presented 
a  briefing  addressing  how  the  facilities  complement  each  other,  how  the 
utilization  of  these  facilities  is  prioritized  and  scheduled,  and  who 
performs  the  scheduling  functions  to  insure  efficient  utilization.  The 
discussion  of  the  final  report  resulted  in  panel  members  being  requested 
to  provide  inputs  on  their  areas  of  particular  interest. 


ARMY  SCIENCE  BOARD 

AD  HOC  SUBGROUP  ON  TESTING  OF  ELECTRONIC  SYSTEMS 


INITIAL  MEETINGS  OF  29-30  JANUARY  1981 

THE  JOHNS  HOPKINS  UNIVERSITY 
APPLIED  PHYSICS  LABORATORY 
LAUREL,  MARYLAND 


Introductory  Remarks 

*  Overview  of  Army  Materiel  Testing 

*  Development  Testing  of  Electronic 

Systems 

*  Operational  Testing  of  Electronic 

Sys  terns 

*  Test  Facilities  at  Fort  Huachuca 

*  Test  Facilities  at  White  Sands 

Missile  Range 

*  TRADOC  Test  Facilities  and  Boards 


Dr.  Mark  Epstein 
Deputy  for  Communications 
and  Target  Acquisition 
OASA(RDA) 


Mr.  J.  P.  Tyler 
DAMA 

Policy,  Plans,  Management 
Division 


Mr.  G.  H.  Banister 

Army  Electronic  Proving  Ground 

Fort  Huachuca,  Az. 


COL  Myron  Motski, 

MAJ  Franklin  Lehman 
OTEA,  Falls  Church,  Va. 


Mr.  G.  H.  Banister 


Mr.  F.  G.  Sebastian 
WSMR,  NM. 


Mr.  G.  D.  Reich 
Fort  Monroe,  Va. 

Dr.  D.  W.  Collier 
Fort  Hood,  Tx. 


Unclassified  Presentation  Material  Available 


ARMY  SCIENCE  BOARD 

AD  HOC  SUBGROUP  ON  TESTING  OF  ELECTRONIC  SYSTEMS 


MEETINGS  OF  3-4  MARCH  1981 

THE  JOHNS  HOPKINS  UNIVERSITY 
APPLIED  PHYSICS  LABORATORY 
LAUREL,  MARYLAND 


Rationale  for  the  Automated 
Tactical  Systems  Test  Bed 
Fort  Hood,  Tx. 


MAJ  R.  L.  Hemphill 
DAMO 

Requirements  Directorate 
Command  and  Control  Division 


The  ASARC  Perspective 
Organizational/Analysis  of  Testing 

DCP  Goals/Thresholds 

Case  Histories 

LTC  J.  E.  Easterbrook 

DAMA 

Command /Control 

Surveillance  Division 

Discussion  of  Army  Program 

Case  History  —  DT/OT 

Dr.  R.  L.  Norwood 

Deputy  for  Air  &  Missile  Systems 
OASA  (RDA) 

Mr.  J.  F.  Bradshaw 

Member,  OASA  (RDA)  Review  Panel 

OSD  Perspective 

Requirements/Decision  Process 

Case  Histories 

Interaction  with  Services 

BG  Eugene  Fox 

Deputy  for  Tactical  Air  &  Land 
Warfare  Systems 

ODD  T&E 

COL  R.  0.  Anderson 

COL  R.  W.  Demon t 

COL  E.  C.  Robinson 

Unclassified  Presentation  Material  Available 
Confidential  Presentation  Material  Available 
Secret  Presentation  Material  Available 
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AD  HOC  SUBGROUP  ON  TESTING  OF  ELECTRONIC  SYSTEMS 


MEETING  OF  3-4  MARCH  1981 

THE  JOHNS  HOPKINS  UNIVERSITY 
APPLIED  PHYSICS  LABORATORY 
LAUREL,  MARYLAND 


*  Navy  C  I  Development  Testing 
Facilities/Testing  Methods 
Case  Histories 


Mr.  C.  T.  Ogata 

Naval  Ocean  Systems  Command 

San  Diego,  Ca. 


***  Navy  Testing  of  Oper.  Communications  Mr.  T.  R.  Evans 

Fleet  Ballistic  Missile  Evaluation  JHU/APL 


Responsive  Threats 
Quantitative  Methods 
Application  to  PERSHING  Program 


**  Testing  and  Evaluation  of  the 
AEGIS  Combat  System 
Ashore  and  At  Sea 
Concept  Evaluation/DT/OT 
Combat  System  Engineering 
Development  Center 
USS  NORTON  SOUND 


*  System  Integration/T&E  for 
Cruiser  "New  Threat  Upgrade" 
Land-Based  Test  Site  —  DT/OT 
Devel.  Support  for  Deployed 
Systems 


CAPT  R.  C.  Beers 
Project  Manager 
Cruiser/Destroyer  Acquisition 

CAPT  G.  R.  Meinig,  Jr. 
Technical  Director 
AEGIS  Shipbuilding  Project 
NAVSEA,  Washington,  D.  C. 


Mr.  J.  W.  Schneider 
JHU/APL 


*  Unclassified  Presentation  Material  Available 
**  Confidential  Presentation  Material  Available 
***  Secret  Presentation  Material  Available 


ARMY  SCIENCE  BOARD 

AD  HOC  SUBGROUP  ON  TESTING  OF  ELECTRONIC  SYSTEMS 

MEETING  OF  18  MARCH  1981 

HEADQUARTERS  TRADOC  COMBINED  ARMS  TEST  ACTIVITY 
FORT  HOOD,  TEXAS 


TCATA/DCSTE 

CofS 

Electronic  Testing  Discussions 

Dir, 

BATD 

Test  Design  Discussions 

P&O/M&A 

TRADOC  Methodology 

ACS, 

M&A 

Instrumentation 

ACS, 

I 
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ARMY  SCIENCE  BOARD 

AD  HOC  SUBGROUP  ON  TESTING  OF  ELECTRONIC  SYSTEMS 

MEETINGS  OF  13-14  APRIL  1981 

UNITED  STATES  ARMY  MISSILE  COMMAND 
REDSTONE  ARSENAL,  ALABAMA 


**  PATRIOT  System  Testing 

BG  Jerry  Max  Bunyard 

BMDSCOM  System  Testing 

Mr. 

Richardson 

*  MICOM  System  Testing  Policy 

Mr. 

Black/Mr.  McCutchen 

*  Software  Testing  Policy 

Mr. 

Ciliax 

*  MICOM  Testing  after  OT  III 

Mr. 

Irvin 

*  PM  Testing  -  HAWK 

Mr. 

Robins 

*  PM  Testing  -  PERSHING 

Mr. 

Tidwell 

*  PM  Testing  -  Air  Defense 

COL 

D.  L.  Wyatt 

Command  and  Control 


*  Unclassified  Presentation  Material  Available 
**  Confidential  Presentation  Material  Available 
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AD  HOC  SUBGROUP  ON  TESTING  OF  ELECTRONIC  SYSTEMS 

MEETING  OF  19  MAY  1981 

US  ARMY  WHITE  SANDS  MISSILE  RANGE 
WHITE  SANDS  MISSILE  RANGE,  NEW  MEXICO 


Software  Testing 

Mr. 

J.  Ellis 

PERSHING  II 

Mr. 

W.  DeBusk 

MLRS  Firing  and  Briefing 

Mr. 

L.  Robinson 

ECM  Testing 

COL 

Mr. 

J.  Pollard, 
B.  Miller 

*  Unclassified  Presentation  Material  Available 
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AD  HOC  SUBGROUP  ON  TESTING  OF  ELECTRONIC  SYSTEMS 

MEETINGS  OF  20-21  MAY  1981 

US  ARMY  ELECTRONIC  PROVING  GROUND 
FORT  HUACHUCA,  ARIZONA 


TRI-TAC  Joint  Test  Element/CTF 
USAEPG  Command  Briefing 
MAINS ITE  Update 
INSBD  Briefing 


COL  Rogers 

COL  Kosmider 

Mr.  G.  H.  Banister 

COL  Dunlap 
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ARMY  SCIENCE  BOARD 

AD  HOC  SUBGROUP  ON  TESTING  OF  ELECTRONIC  SYSTEMS 

MEETINGS  OF  9-10  JULY  1981 

THE  JOHNS  HOPKINS  UNIVERSITY 
APPLIED  PHYSICS  LABORATORY 
LAUREL,  MARYLAND 


TECOM  Workload  Planning 


Mr.  Louis  Teletski 
Aberdeen  Proving  Ground 
Aberdeen,  Maryland 


TECOM  Initiative 
Master  Resource  Programming 


Mr.  George  Schroeter 
Aberdeen  Proving  Ground 
Aberdeen,  Maryland 
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I.  Missions  of  US  Army  Test  and  Evaluation  Command  Facilities. 

A3ERDEEN  PROVING  GROUND 

Mission.  To  conduct  testing  of  tank  and  small  arms  weapons  systems, 
ancillary  munitions  and  components,  survey  and  target  acquisition  material, 
armor  plate,  combat,  general  and  special  purpose  vehicles,  combat  engineer 
and  troop  support  equipment. 

DUGWAY  PROVING  GROUND 

Mission.  To  conduct  testing  of  chemical  weapons,  chemical/biological 
defense,  flame,  incendiary  and  smoke  munitions  systems  and  provide  technical 
assessments  of  Foreign  Biological  Threats;  manage  and  execute  the  DoD  Joint 
Chemical-Biological  Contact  program. 

ELECTRONIC  PROVING  GROUND 

Mission.  To  conduct  testing  of  communications-electronics ,  optical/ 
electro-optical  systems,  signal  intelligence,  electronic  warfare  systems 
and  other  electronic  material. 

JEFFERSON  PROVING  GROUND 

Mission.  To  conduct  production  acceptance,  pest  production,  product 
improvement,  malfunction  investigation,  propellant  assessment  and  master 
calibration  tests  of  ammunition  and  ancillary  components. 

WHITE  SANDS  MISSILE  RANGE 

Mission.  To  conduct  testing  of  rocket  and  guided  missile  systems, 
ancillary  guidance/navigation  systems,  air  defense  fire  distribution 
systems,  laser  weapons  systems,  and  other  designated  material. 

YUMA  PROVING  GROUND 

Mission .  To  conduct  testing  of  tube  artillery,  aircraft  armament  and 
air  delivery  systems,  air  movable  and  mobility  equipment  and  the  natural 
desert  environmental  phases  of  developmental  test  of  all  classes  of  defense 
material. 


f 


D-l 


AVIATION  DEVELOPMENT  TEST  ACTIVITY 


Mission.  To  conduct,  evaluate  and  report  on  test  elements  of 
government  developmental  and  product  improvement  testing  and  reporting 
on  contractor  test  elements  of  the  Single  Integrated  Development  Test 
Cycle  (SIDTC)  of  aircraft,  aircraft  components  (time-between  overhaul, 
time-between  inspection)  and  aircraft  related  support  equipment. 

COLD  PJZGIONS  TEST  CENTER 

Mission.  To  conduct  cold  regions  environmental  testing  of  all 
classes  of  material/systems. 

TROPIC  TEST  CENTER 

Mission.  To  conduct  tropic  environmental  testing  of  all  classes 
of  material/systems. 


I I .  Missions  of  US  Army  Training  and  Doctrine  Command  Facilities. 

US  Army  Combat  Developments  Experimentation  Command  Mission.  To  partici¬ 
pate  in  the  combat  and  training  development  process  by  conducting  field 
experimentation  to  provide  high  resolution,  accurate  data,  collected  in 
an  operational  environment,  necessary  for  improvement  of  combat  effective¬ 
ness  . 


a.  USACDEC  supports  the  US  Army  Training  and  Doctrine  Command  (TRADOC) 
and  the  Operational  Test  and  Evaluation  Agency  (OTEA)  in  operational  test¬ 
ing  and  is  responsible  for  test  design  and  execution  of  small  scale  force 
development  testing  and  experimentation  (FDTE) . 

b.  USACDEC  plans,  designs,  and  procures  instrumentation  systems  for 
the  collection  of  data  and  conducts  methodology  and  feasibility  tests 
to  develop  rationale  for  validation  of  data  collection  means. 

c.  USACDEC  experiments  are  typified  by  force--on-force ,  statistically 
reliable  tests  utilizing  real  time  casualty  assessment. 

US  Army  TRADOC  Combined  Arms  Test  Activity  Mission.  To  plan  and  conduct 
operational  and  force  development  test u  and  evaluations  in  support  of 
TRADOC  combat  development  and  training  development  programs  and  to  develop 
instrumentation  concepts  and  exportable  advanced  technology  instrumentation 
to  support  testing,  field  exercises,  and  training.  The  scope  of  TCATA 
testing  includes  combined  arms,  combat  service  support,  and  logistics  field 
testing  relating  to  material,  tactics,  organization  and  doctrine;  command 
systems  testing  relating  to  intelligence,  integration,  electronic  warfare, 
and  tactical  ADP;  and  training  developments  evaluations  of  programs, 
simulators  and  devices. 


TEST  BOARDS 


US  Army  Armor  and  Engineer  Board 

US  Army  Airborne  Board 

US  Army  Intelligence  and  Security  Board 

US  Army  Air  Defense  Board 

US  Army  Aviation  Board 

US  Army  Communications-Electronics  Board 
US  Army  Field  Artiller'-  Board 
US  Army  Infantry  Board 

Mission:  The  test  boards  are  assigned  the  following  missions  in 
their  respective  commodity  areas: 

a.  Flan,  conduct,  and  report  on  operational  and  other  user  testr. 

b.  Participate  in  other  testing  as  directed. 

c.  Provide  advice  and  guidance  on  test  matters  to  combat,  training, 
and  material  developers,  other  services  and  private  industry. 

d.  Conduct  other  tests  and  selected  specific  evaluations  as 
directed  by  CDR  TRADOC. 


III.  Glossary  of  Testing  Terris 


a.  Testing  Terms: 


b.  Other  Terms: 


ADVT  -  Advance  Development  Verification  Test 

C  -•  Contractor 

DT-I  --  Development  Test  -  I 

DT-II  -  Development  Test  -  II 

EDT  -  Engineer  Design  Test 

FOE  --  Follow-on  Evaluation 

G  -  Government 

OT-T  -  Operational  Test  -  I 

OT-II  -  Operational  Test  -  II 

POT  -  Frototype  Oualif ication  Test 

PVT  -  Production  Validation  Test 

ADTA  -  Aviation  Development  Test  Activity 

AEFA  -  Aviation  Engineer  Flight  Activity 

ARRADCCK  -  Armament  Research  and  Development 
Command 

ATSTB  -  Automated  Tactical  System  Test  Bed 

CDEC  -  Combat  Development  Experimentation 
Command 

CECOM  -  Communication  Electronics  Command 

CSL  -  Chemical  Systems  Laboratory 

FORSCOM  -  Forces  Command 

TACON  -  Tank  Automotive  Command 

TCATA  -  TRADOC  Combined  Arms  Test  Activity 

WS?fR  -  White  Sands  Missile  Range 
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IV.  Nummary  of  Types  of  Testing  Conducted  at  Facilities/Activities 
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DT-Development  Test 
0T-  Operational  Test 

PVT-G  -  Production  Verification  Test,  Govt 
PIT  -  Product  Improvement  Test 
FOE  -  Follow-on  evaluation 


Progression  of  Types  of  Tests  Through  Army  Facilities/Activities 

A.  Aviation  Systems 

B.  Chemical /Tiologicnl/Incendiary/Smoke 

C.  Artillery  Systems 

D.  Tank/Automotive  Systems 

E.  Missile  Systems 

3 

F.  Cl  Systems 


G.  Munitions 


AVIATION  SYSTEMS 


Contractor  Facility 


Aviation  Engineering  Flight  Activity  (AEFA) 
Contractor  Facil ity/AEFA 


Aviation  Development  Test  Activity  (ADTA) 


Contractor  Facility/AEFA 

AEFA 

Contractor  Facility  or  Government  Facility 

ADTA,  Yuma/Ft.  Huachuca 


CDEC/TCATA 


PVT-C 


-  Contractor  Facility 


B.  CHEMICAL/BIOLOGICAL/INCENDIARY/SMOKE 


EDT-C  j  -  Contractor  Facility/CSL 

EDT-G  ^  -  Dugway  Proving  Ground 

ADVT-C  A  -  Contractor  Facility/CSL 

ADVT-G ^  -  Dugway  Proving  Ground 
Normally  combined  with  DT-I  at  Dugway 


-  —  —  - 

^  PQT-C  ^  -  Contractor  Facility,  CSL 

PQT-G  ^  -  Dugway  Proving  Ground 

-  FORSCOM  Installation/Firing  using  Chemical  Simulants  at  DPG 


PVT-C  J-  Contractor  Facility/IXigway 
PVT-G  A  -  Dugway 


-  FORSCOM  Installation/Dugway 
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C.  ARTILLERY  SYSTEMS 


-  Contractor  Facility 


-  Aberdeen/Yuma  ^roving  Ground 

-  Contractor  Facility/ARRADCOM 

-  Aberdeen/Yuma/WSMR 


Aberdeen/Yuma/Ft.  Sill  Field  Artillery  Board 


-  Contractor  Facility/ARRADCOM 

-  Aberdeen/Yuma 

-  Contractor  Facility/ARRADCOM 

-  Aberdeen/Yuma/WSMR 


FORSCOM  Installation/WSMR 


-  Contractor  Facility 

-  Aberdeen/Yuma/WSMR 


FORSCOM  Installation  (Ft.  Sill,  Ft.  Hood) /WSMR  for 

missiles  and  rockets 


GD-A122  26 5 
UNCLASSIFIED 


ARNV  SCIENCE  BOARD  AD  HOC  SUBGROUP  ON  TESTING  OF 
ELECTRONIC  SVSTEHS(U)  ARNV  SCIENCE  BOARD  WASHINGTON  DC 
07  SEP  81 

F/G  14/2 
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B.  CHEMICAL/BIOLOGICAL/INCENDIARY/SMOKE 


-  Contractor  Facility/CSL 


-  Dugway  Proving  Ground 


-  Contractor  Facility/CSL 


-  Dugway  Proving  Ground 


i  -  Normally  combined  with  DT-I  at  Dugway 


-  Contractor  Facility,  CSL 

-  Dugway  Proving  Ground 


-  Contractor  Facility,  CSL 


-  Dugway  Proving  Ground 


-  FORSCOM  Installation/Firing  using  Chemical  Simulants  at  DPG 


-  Contractor  Facility/Dugway 


-  Dugway 


-  FORSCOM  Installation/Dugway 
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C.  ARTILLERY  SYSTEMS 


-  Contractor  Facility 


-  Aberdeen/Yuma  Proving  Ground 

-  Contractor  Facility/ARRADCOM 


-  Aberdeen/Yuma/WSMR 


-  Aberdeen/Yuma/Ft.  Sill  Field  Artillery  Board 


/  N 

EDT-C  }  -  Contractor  Facility/ARRADCOM 


Aberdeen/Yuma 


-  Contractor  Facility/ARRADCOM 


Aberdeen/Yuma/WSMR 


FORSCOM  Installat ion/WSMR 


-  Contractor  Facility 

-  Aberdeen/Yuma/WSMR 


FORSCOM  Installation 


(Ft.  Sill,  Ft.  Hood) /WSMR  for 

missiles  and  rockets 
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D.  TANK/AUTOMOTIVE  SYSTEMS 


-  Contractor  Facility 


-  Aberdeen/Yuma  Proving  Grounds 


-  Contractor  Facility/TACOM 


-  Aberdeen/Yuma 


Aberdeen/Yuma 


)- 


Contractor  Facility 

-  Aberdeen/Yuma 


FORSCOM  Installation 
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E.  MISSILE  SYSTEMS 


Contractor  Facility/Missile  Command 

WSMR 

-  Contractor  Facility/Missile  Command 

-  WSMR 

-  Contractor  Facility/Missile  Command 

-  WSMR/Missile  Command 

-  Contractor  Facility/Missile  Command 

-  WSMR 

at  FORSCOM  Installation 

Contractor  Facility/Missile  Command 

-  WSMR 


FOE 


-  (Same  as  OT-II) 


F.  C3I  SYSTEMS 
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G.  MUNITIONS 


Contractor  Facility 


-  Aberdeen/Yuma  Proving  Grounds 


ADVT-C  ^  -  Contractor  Facility/ARRADCOM 


ADVT-G  A  -  Aberdeen/Yuma  Proving  Grounds/WSMR 


Aberdeen/Yuma  Proving  Grounds 


-  Contractor  Facility/ARRADCOM 


-  Aberdeen/Yuma 


-  Contractor  Facility/ARRADCOM 


-  Aberdeen/Yuma/WSMR 


FORSCOM  Installation 


PVT-C 


-  Contractor  Facility/Jefferson  Proving 
Ground /WSMR 


PVT-G  )  -  Jef f erson/WSMR 


FORSCOM  Installation/WSMR  for  guided  Projectiles 


PROCURE  AND  OPERATE  ECM  TEST  TRANSMITTERS 


SPECIFICATIONS  AND  TEST 


SOFTWARE  DESIGNER/CODERS 


SYSTEM/SOFTWARE  DEVELOPMENT  CYCLE 


PROGRAM 

UNITS 


PATRIOT  -  CONCEPT  POPULAR  DURING  1960's 


REVISED  SOFTWARE  DEVELOPMENT  PLAN 


PROGRAM 
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ADEQUATE  RESOURCES  ALLOCATED 
SOFTWARE/HARDWARE  RECEIVING  EQUAL  EMPHASIS 
MAKES  MAXIMUM  USE  OF  ALL  TEST  DATA 


IMPROVEMENTS  TO  ELECTRONIC  SYSTEMS  TESTING  13  April  1981 


L9 

>- 

OO 

LU 

ac 

ca 

2 

LU 

oc 

ac 

<c 

<C 

e 

o 

F— 

OO 

K 

OO 

OO 

1 — 

Of 

L9 

o 

OO 

LU 

L9 

O 

«— A 

>- 

OO 

O 

OC 

F- 

oo 

al 

LU 

or 

CU 

<c 

pa 

o  - 

Q_ 

CL 

*"*T> 

_l  F— 

>- 

F— 

— 1 

oo 

lu  ra 

h— 

3— 

OO 

<c 

>■  o 

O 

»— i 

LU 

3»* 

lu  oc 

F— 

h— 

LU 

ca  - 

ca  oo 

o 

F— 

ra 

OO 

* 

ca 

ca  5 

LU  o 

Q_ 

O  LU 

O 

z 

LU  CXC 

oc  oc 

OC  OO 

<c 

0_>  09 

<c  =c 

ca 

LU  <C 

oo  - 

<c  o 

3c  1 — 

LU 

T 

gF" 

F— 

_J  PC 

F— 

OO 

CL.  CU 

OO  <£ 

oo 

CU  CU 

LU  F- 

LU 

3E 

LU 

ca  z 

UJ  09 

F— 

<--»  LU 

OO  LU 

LU 

oo  o 

oo  o 

09  O 

( — 

O 

OO  — • 

z  oc 

LL  OO 

• 

LU  F— 

O  CU 

F— 

—  OO 

o  — 

OC  C9 

L_>  U_> 

CJ 

O 

ca  lu 

OO 

uj  z 

O  =9 

N 

Z  09 

_ I  z 

PQ  *— * 

oc  ca 

ca  ■— 

I — 

=a  <c 

o  o 

LU  Q_ 
OC  O 


<3  > 
OJ  LU 


Q_  O 
OC 
h-  CL 
OO 

LU  LU 
F—  O 
I 

X  09 


OO  — 
LU  09 


>  ca 
o  z 


*9 

I—  OC 
jju  UJ 

F-  >- 


z  o 

«— •  LU 


CL  F- 
5  LU 
Q- 

lu  ca 


CC  U_> 


O  CJQ 
OU 

cag 


— <  L3 

—I  o 
• — .  ac 

CO  CL 
OO  LU 
>  F— 


E-9 


PLAN  FOR  INDEPENDENT  EVALUATION  OF  SYSTEM  (TO  INCLUDE  SOFTWARE)  MUST  BE 
INITIATED  AT  THE  EARLY  PART  OF  FSED. 


h— 

m 

LU 

CO 

s 

LU 

<_> 

u 

eg 

eg 

■ 

o 

<3: 

co 

• 

LU 

LU 

eg 

o 

> 

>  t 

»■  » 

H— 

_ 1 

LU 

c_> 

i 

O 

^3»» 

(=3 

h- 

o 

O 

CO 

eg 

Q_ 

a 

eg 

LU 

CO 

Q_ 

O 

o 

»— 

a. 

a 

a_ 

o  - 

«a: 

o 

CO  1 — 

to 

eg 

<C  CO 

Q_ 

o 

r 

LU  (-> 

(=3 

C=> 

_ 1 

LU 

OQ 

— •  CO 
CO  CO 

LI— 

CO  LU 

LU 

H 

O  — 1 

CO 

CL. 

<C 

y 

h— 

O 

*—  <c 

C2J 

s 

lu  s ti 

LU 

C=3 

a 

►—  LU 
X  K— 

LU 

LU  CO 

CO 

CQ 

>- 

LU 

z;  co 

eg 

t— 

a. 

s:  u_ 

LU 

—  o 

cs 

c_) 

LU 

eg 

Q 

L-> 

ft— 4 

1 

>- 

O 

£ 

CO  — 1 

_ 1 

eg 

Hj 

LU  Cg 

§ 

>  LU 

s 

*— ?  1— 

<c 

►— 

Cg  h— 

Q_ 

eg 

a  lu 

<C 

o 

eg 

L_) 

a_ 

I—  w 

Q_ 

CO 

LU  LO 

5 

CO 

I—  z 

LU 

LU 

LU  1— 

LU 

eg 

r^i  co 

> 

»■  ■« 

— •  LU 

O 

_l  1— 

eg 

C2J 

a_ 

LU 

eg 

=)  — 

• 

E-10 


THE  JOHNS  HOPKINS  UNIVERSITY 

applied  physics  laboratory 

LAulif:  Ma  i  a  Nil 


JFB-P80-005 
8  September  1980 


A.  R.  Eaton 
J.  F.  Bradshaw 

An  Approach  for  Testing  High  Technology  Electronic  Systems 

(a)  Memorandum  for  Assistant  Secretary  of  the  Army  (RDA) 
dated  18  March  1980  from  General  John  W.  Vessey,  Jr., 
Subject:  Testing  of  High  Technology  Electronic  Systems 

(b)  Memorandum  for  Vice  Chief  of  Staff,  Army 
dated  9  June  1980  from  Dr.  Percy  A.  Pierre, 

Subject:  Testing  of  High  Technology  Electronic  Systems 

Reference  (a)  states  an  Army  concern  about  its  ability  to 
perform  adequate  developmental  testing  for  some  of  its  advanced  weapon 
systems.  It  is  suggested  that  the  Army  does  not  have  a  good  capability 
to  load  its  advanced  automated  weapon  systems  with  realistic  battlefield 
conditions  to  perform  thorough  developmental  tests.  Sophisticated  and  complex 
battlefield  environmental  conditions  cannot  be  adequately  simulated  to  provide 
the  density  and  fidelity  required  to  stress  these  new  systems  during  the 

development  period.  This  results  in  costly  operational  tests  to  be  run  to 

discover  deficiencies  that  should  have  been  found  earlier  in  the  development 
process.  Crucial  operational  performance  information  is  lost  because  precious 
OT  testing  time  is  primarily  used  to  isolate  and  debug  developmental  problems. 
Therefore,  a  comprehensive  cost  effective  developmental  test  tool  should  be 
developed  to  relieve  this  problem. 

There  is  a  demonstrated  system  approach  that  has  been  used  in 

other  programs  that  can  help  cope  with  this  testing  problem.  It  uses  a 

computer-based  test  tool  to  drive  the  operational  system.  This  test  tool  has 
the  capability  to  read  and  record  all  digital  input  and  output  data  from  the. 
operational  system.  For  example,  it  would  prerecord  forward  observer  data, 
battery  computer  system  data,  fire  support  officer  data,  air  observer  data 
and  general  support  battalion  data  so  that  this  information  could  be  used 
to  drive  the  operational  system  under  test  in  a  controlled  manner.  This 
digital  recording  technique  could  record  all  inputs  to  the  operational  system 
in  heavy  background  load  and  ECM  environments,  thereby  providing  a  ready 
library  of  high  volume  inputs  for  a  variety  of  background  situations.  Since 
the  environments  would  be  recorded  digitally,  tactical  situations  could  be 
replicated  in  a  very  controlled  fashion. 


To: 

From: 

Subj  ect : 
References: 


Mr.  Bradshaw  is  an  employee  of  JHU/APL;  he  is  a  member  of  a  PATRIOT  Program  Review  Panel 
established  by  the  Assistant  Secretary  of  the  Army  (RD&A). 
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Complementing  this  background  recording  capability,  the 
test  tool  would  also  have  an  automatic  and  manual  interactive  battlefield 
situation  scenario  scripting  capability  that  could  be  operated  with  or 
without  the  recorded  background  environment  data.  This  would  allow  for  a 
variety  of  battlefield  situation  scenarios  to  be  overlaid  upon  a  variety 
of  background  environments,  all  of  which  would  be  completely  repeatable 
since  the  data  is  in  digital  format.  This  system  would  also  have  the 
built-in  capability  of  tracing  message  data  to  its  original  source  and 
assessing  whether  or  not  the  weapons  operator  correctly  performed  his  duties. 
A  system  of  this  type  would  provide  the  density  of  data  required  and  also 
control  the  test  process  so  that  when  operational  computer  program  faults 
occur  the  situation  could  be  repeated  in  order  to  trap  and  isolate  software 
errors. 


Another  significant  feature  of  this  test  system  would  be  the 
manual  real  time  operator  interactive  scripting  capability.  This  would 
allow  a  testing  officer  to  change  in  real  time  the  "canned"  battlefield 
scenarios  and  thereby  stress  the  operational  system  in  particular  areas, 
if  desired.  This  particular  feature  has  some  rather  significant  payoffs 
for  operator  training  during  the  early  phases  of  operational  testing. 

It  should  be  noted  that  this  approach  for  a  test  tool  does 
not  test  the  input  data  hardware  of  the  weapon  system,  but  is  used  to  drive 
the  operational  system  automatic  processing  capability  and  record  operator 
interaction  and  user  output  portions  from  the  system.  Sensor  hardware 
testing  would  be  conducted  separately.  This  test  tool  has  the  added  feature 
in  that  it  can  also  be  used  after  operational  tests  are  completed.  The 
recording  portion  of  the  system  would  be  activated  during  the  operational  test 
and  all  interface  data  recorded  in  real  time.  When  operational  tests  are 
complete  and  weapon  system  upgrades  have  been  made,  this  recorded  data  could 
be  played  back  into  the  weapon  system  to  verify  improvements.  The  complexities 
of  the  operational  test  are  not  lost  and  can  be  used  over  and  over  again  to 
restress  the  weapon  system,  thus  saving  the  enormous  cost  of  repeating  the 
subsequent  operational  test.  This  system  provides  the  means  of  developing 
and  testing  complex  weapon  systems  without  the  redundant  expense  of  using 
actual  attack  aircraft  or  field  units  over  and  over  again.  Such  a  test  tool 
can  exercise  the  weapon  system  in  a  controlled  repeatable  manner  to  a  much 
greater  extent  than  is  now  possible. 

RECOMMENDED  APPROACH 


There  are  several  questions  that  should  be  considered  before 
embarking  on  a  test  tool  development  process.  These  are: 

1.  What  are  the  specific  goals  and  objectives  for  this  test  tool? 

2.  Is  this  test  tool  to  be  considered  universal  for  multi-systems 
tests  or  just  for  individual  weapon  systems  tests? 
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3.  Are  there  any  written  test  tool  system  performance  and 
compatibility  requirements? 

4.  Is  training  to  be  considered  a  feature  of  the  test  tool? 

5.  At  what  levels  should  the  test  tool  operate  -  division, 
battalion,  battery,  fire  unit? 

6.  Are  there  adequate  scenario  descriptions  to  generate 
battlefield  conditions? 

7.  Are  there  adequate  statistical  measures  to  modulate 
battlefield  data  as  a  function  of  environment  -  natural  or  manmade? 

8.  Is  the  Army  data  collection  philosophy  consistent  with 
automated  data  reduction  processing? 

9.  Are  there  independent  test  teams  identified  and  working 
during  the  development  phase?  If  yes,  are  they  independently  funded? 

10.  Are  test  system  data  collection  requirements  established 
prior  to  the  design  of  the  operational  system? 

11.  Does  the  Army  have  a  catalog  of  available  test  tools? 

12.  Has  the  Army  established  guidelines  regarding  a  development- 
to-test  cost  ratio? 

13.  Are  the  test  tools  developed  under  the  same  discipline  as 
the  operational  program?  (Are  we  using  test  tools  that  haven't  been  tested?) 

14.  Does  the  Army  have  an  independent  audit  team  that  conducts 
software  audits  during  the  software  development  process  to  establish  system 
baseline  descriptions? 

15.  Does  the  current  development  process  support  intermediate 
independent  developmental  testing? 

It  is  recommended  that  a  panel  of  experts  be  convened  to  formulate 
answers  to  these  questions  and  begin  formulating  a  requirements  document  for 
the  proposed  test  tool. 


F.  Bradshaw 
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Distribution: 


A.  R. Eaton 

B. D. Dobbins 
R.L.Ely 


19  September  1980 


To:  A.  R.  Eaton 

From:  N.  A.  Begovich 

Subject:  Testing  of  High  Technology  Electronic  Systems 


Large  electronic  decision  systems  (air  defense,  C^,  etc.)  testing 
requires  the  use  of  computer  simulation  of  the  real  world  environment  to 
fully  load  and  test  the  system.  Live  testing,  at  best,  can  only  give  confi¬ 
dence  the  test  environment  is  a  faithful  presentation  of  the  real  world  at 
the  live  testing  load  level.  The  test  environment  computer  program  develop¬ 
ment  is  as  difficult  a  task  as  the  system  operational  program  development. 
However,  the  hardware  used  in  the  test  environment  should  be  commercial  or 
off-the-shelf  so  that  the  software/hardware  change  problem  is  non-existent. 

o  The  DD963  and  the  AN/SYS-1  programs  are  examples  of  successful 
test  environment  computer  program  developments  that  provided  a 
"real  world"  tactical  environment  for  system  testing  and  for 
crew  training. 

o  The  development  of  a  large  electronic  decision  system  should  be 

paralleled  with  the  development  of  the  test  environment  that  will 
permit  full  tactical  load  system  testing.  The  system  and  the 
test  environment  developments  should  be  performed  by  different 
contractors.  This  will  insure  that  the  test  environment  will 
reflect  the  system  requirements  and  not  the  system  contractor  con¬ 
cept  of  the  requirement.  Periodic  program  reviews  with  both  the 
system  and  test  environment  contractors  participating  should 
prevent  any  system  design/requirements  incompatibilities. 

o  Army  organizations,  the  particular  school  reflecting  the  user 

requirements  and  OTEA,  should  participate  in  the  test  contractor 
design  reviews  so  that  the  test  environment  reflects  the  user 
requirements.  This  forces  the  user  to  quantify  his  requirements 
during  the  system  development. 


Dr.  Begovich  is  a  private  consultant;  he  is  a  member  of  a  PATRIOT  Program  Review  Panel 
established  by  the  Assistant  Secretary  of  the  Army  (RD&A). 
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o  The  parallel  development  of  the  system  and  the  test  environment 
(see  Figure  1)  has  the  following  advantages: 

o  Minimum  time  and  resources  expenditure  in  fielding 
a  new  system 

o  Minimum  live  target  (aircraft,  troops,  etc.)  testing 

o  Realistic  system  training  facility 

o  Repeatable  test  scenarios  for  system  correction  and 
crew  training 

o  System  requirements  frozen  during  and  not  after  system 
development . 

The  Army's  present  serial  system  development  process  is  shown  in 
Figure  2.  The  user  participates  initially  in  defining  the  system  requirements. 
After  the  system  is  developed,  the  user  again  gets  involved  in  conducting  the 
system  test.  Unfortunately,  sufficient  time  has  elapsed  that  the  user  has  a 
different  concept  on  what  the  system  should  do  resulting  in  the  system  being 
returned  to  the  contractor  for  correction  of  deficiencies.  This  cycle  o 
test,  correction  and  retest  has  been  repeated  in  some  systems  (Q-73,  TACFIRE) 
developments  so  many  times  that  when  the  system  finally  passes  the  user  test, 
the  hardware  is  obsolete.  The  Army  then  has  the  choice  of  procuring  and 
deploying  a  system  having  a  hardware  design  that  is  at  least  ten  plus  years 
old  or  not  deploying  any  system. 

A  new  computer  hardware  generation  is  occurring  every  three  plus 
years.  The  threat,  consequentially,  is  also  changing  on  a  similar  time 
cycle.  The  Army's  serial  system  development  process  that  has  a  manyfold 
longer  time  period  will  always  produce  obsolete  systems. 

To  insure  timely  success  in  the  Parallel  System  Development  process 
the  following  are  absolute  requirements. 

o  The  test  environment  development  should  be  performed  by  a 

contractor  and  not  a  government  agency.  The  government  must 
resolve  problems  between  the  system  and  test  environment 
developers  and  cannot  be  a  "judge  and  one  of  the  contestants". 
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The  test  environment  development  should  be  for  a  particular 
program  and  not  a  universal  test  bed.  A  universal  test  bed 
will  result  in  a  development  for  its  own  end  purpose  instead 
of  being  optimized  to  test  a  particular  system. 

All  the  test  environment  hardware  should  be  "off-the-shelf 
so  that  the  hardware  baseline  is  fixed  during  the  test 
software  development  process. 


N.  A.  Begovich 
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Figure  1  Parallel  Systems  Development  Process 


Figure  2  Serial  Development  Process 
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